acceptodds
Withdrawn from ICLR 2027

UTPS: Universal Taylor Proxy Scoring for Sparse Multimodal Low-Rank Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adapting Multimodal Large Language Models (MLLMs) to downstream tasks is computationally expensive. While Low-Rank Adaptation (LoRA) reduces trainable parameters, applying it uniformly across all layers is inefficient and can disrupt pretrained representations. We propose Universal Taylor Proxy Scoring (UTPS), a lightweight, training-free method that identifies the most task-relevant layers for sparse adaptation using only a single forward-backward pass. UTPS integrates gradient-weight sensitivity, activation-aware scaling, and parameter-density normalization to prioritize layers with the highest performance-per-parameter. Guided by UTPS scores, we design a sparse dynamic multi-expert LoRA architecture with multimodal-aware budget allocation. Experiments on InternVL2-2B, LLaVA-1.5-7B, InternVL2-8B, Qwen2.5-VL-3B, and LLaMA-2-7B demonstrate that UTPS, using only 50% of LoRA parameters, closely approaches or matches full LoRA performance across diverse multimodal and text-only benchmarks. Our work thus enables efficient, targeted adaptation of large models without compromising accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zsqiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Universal Taylor Proxy Scoring (UTPS), a lightweight, training-free method that identifies the most task-relevant layers for sparse adaptation using only a single forward-backward pass. UTPS integrates gradient-weight sensitivity, activation-aware scaling, and parameter-density normalization to prioritize layers with the highest performance-per-parameter. The results seems good, but several inconsistencies make the main claims hard to verify.

Critical strengths
  1. Selecting useful layers for LoRA is a practical idea, and the method is simple.

  2. The experiments cover several tasks and models, with ablations, random selection, and different budgets.

Critical weaknesses
  1. The layer counts in Fig. 2 need clarification. InternVL2-2B has 24 vision and 24 language Transformer layers, yet the figure reports 48 selected layers at a 50% budget, with some counts exceeding 24 per modality. If these numbers refer to modules or expert units, please define the counting unit and explain how it relates to the stated budget.

  2. Selecting 50% of the layers does not necessarily mean using 50% of the LoRA parameters. Vision and language layers have different parameter costs. Tab. 4 only gives percentages, so the claimed budget matching across methods cannot be checked.

  3. Some p-values in Tab. 7 appear inconsistent with the reported means and standard deviations. For example, the Flickr30k results imply a two-sided paired t-test p-value of at least about 0.057 under the stated three-run setup, rather than 0.02. Tab. 1 also reports best results, while the reproducibility statement says averages.

  4. The explanation of Taylor-W is not supported by its formula. Taking the norm of the gradient–weight product removes signs and does not measure overall directional alignment. It therefore cannot support the claimed distinction between helpful and harmful updates. Its connection to actual LoRA gains also needs evidence.

Questions for authors
  1. What does “layer” mean in Fig. 2? Please explain why 48 layers are selected at a 50% budget and why some counts exceed 24 per modality.

  2. Please provide the actual trainable parameter counts for each method and task. How do you ensure equal parameter budgets when vision and language layers have different costs?

  3. Please provide the individual results behind Tab. 7 and explain how the p-values were calculated, including whether the tests were one-sided or two-sided. Are the results in Tab. 1 best values or averages?

  4. How does Taylor-W distinguish helpful from harmful updates when the norm removes signs? Is there direct evidence that its scores predict actual layer-wise LoRA gains?

Reviewer_xAvBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UTPS, a training-free layer-selection method for parameter-efficient LoRA adaptation, achieving competitive performance with only 50% of the trainable parameters. The paper is clearly written and easy to follow. However, the motivation and advantages over existing PEFT methods are not sufficiently justified. Moreover, the lack of comparisons with relevant methods and detailed efficiency analysis limits the evaluation of the proposed approach.

Critical strengths
  1. The paper is clearly written and easy to follow, with a well-structured presentation
Critical weaknesses
  1. The paper proposes a layer-selection strategy for LoRA adaptation, but the motivation for this design is not sufficiently justified. If the goal is to improve parameter efficiency, existing methods such as LoSA and VeRA have already explored parameter-reduction strategies. If the goal is to enhance task-specific adaptation, numerous studies have investigated combining LoRA with Mixture-of-Experts (MoE) architectures. The paper does not clearly explain how its proposed approach addresses limitations of these existing methods.
  2. The experiments lack comparisons with these relevant methods and do not report essential efficiency metrics, such as the number of trainable parameters, training time, and peak GPU memory usage. Without these comparisons, it is difficult to assess the practical benefits and relative advantages of the proposed method.
Questions for authors
  1. Could the authors further clarify the primary motivation for the proposed layer-selection strategy and explain its advantages over existing parameter-efficient methods, such as LoSA, VeRA, and MoE-based LoRA approaches?
Reviewer_sXCVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UTPS for sparse LoRA layer selection in multimodal models. It combines gradient–weight sensitivity, activation magnitude, and parameter-density normalization to allocate adaptation budgets across vision and language modules. Experiments cover six multimodal tasks, component ablations, budget sweeps, and additional model backbones.

Critical strengths
  1. The method addresses a practical layer-allocation problem. At the reported 50% LoRA budget, UTPS exceeds budget-matched LoSA on all six point estimates and approaches full LoRA in the reported aggregate (85.04 versus 85.80). Three-seed comparisons are significant on five tasks.

  2. The evaluation includes component ablations, random layer selection, budget sweeps, and transfers to other multimodal and text-only models. The reported scoring overhead is below 0.6% of training time.

Critical weaknesses
  1. The score is weakly justified for additive LoRA updates. Taylor-W relies on pretrained weight magnitudes, although actual low-rank updates are additive; the authors acknowledge its heuristic nature. Some ablation gaps are small relative to seed variation. LoRA-aware gradient proxies and controls separating scoring from allocation are missing.

  2. Experimental reporting needs clarification. Table 1 reports “Top-1 over three seeds,” which appears to indicate best-seed results and exceeds the three-seed means in Table 7, while seed statistics for full LoRA and D-MoLE are absent. The 50% budget is allocated by layer count without tabulated absolute parameter counts. Table 5 implies about 10–22× higher per-sample scoring throughput than training. The claimed minimum vision budget also contradicts Section 3.4, and multi-task interference reduction is not directly evaluated.

Questions for authors
  1. Are Table 1 values best-seed results? Can the authors report mean ± SD and absolute trainable-parameter counts for the budget-matched methods, and explain the roughly 10–22× scoring-versus-training per-sample throughput gap implied by Table 5?

  2. Under matched parameter budgets, how does UTPS compare with LoRA-aware gradient proxies or simple structured layer selection? Can scoring and modality-allocation effects be isolated, for example by swapping the scoring and allocation rules with D-MoLE?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.