acceptodds
Withdrawn from ICLR 2027

AdaSparse-PRL: Progressive Reinforcement Learning Fine-tuning with Adaptive Sparse-Aware for Multimodal LLM

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) excel in cross-modal understanding, fine-tuning is inefficient due to the interdependence of heterogeneous visual and textual features within a shared latent space. To address these issues, we propose AdaSparse-PRL, a progressive reinforcement learning fine-tuning framework with adaptive sparse awareness for MLLMs. Specifically, we first introduce a fine-grained implicit cross-modal decoupling mechanism, which separates latent visual and textual representations at a finer semantic level and reduces interference among heterogeneous modalities during multimodal alignment. Then, an adaptive fine-tuning strategy with dynamic sparsity awareness is designed to identify task-relevant parameters and modal-sensitive activation patterns, thereby eliminating redundant computations and improving training efficiency. Finally, we develop a progressive reinforcement learning method with preference alignment, where the model is optimized through staged reward guidance and gradually aligned with human preferences, thereby improving multimodal instruction-following performance. Extensive experiments on multimodal benchmarks demonstrate that AdaSparse-PRL can increase training speed by more than 1.6x while maintaining accuracy, and reduce computing and memory overhead by more than 30%. Our code and models will be publicly available.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multimodal Large Language Models, Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PTCJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AdaSparse-PRL, an efficient multimodal fine-tuning framework that combines: (1) implicit cross-modal decoupling of visual and textual representations into shared and modality-specific components, (2) input-adaptive sparse parameter updates based on gradient and representation statistics, and (3) progressive reinforcement learning involving reward-model training, scheduled PPO optimization, and sparsity-preserving alignment. The method is evaluated on MM-Vet, MMBench, and VQAv2 using several Qwen-VL model sizes. The paper reports improvements over PPO, RFT, LoRA, QLoRA, GRPO, and other fine-tuning methods while claiming substantial reductions in training time, FLOPs, and memory.

The problem is important, but the proposed algorithm is not described consistently enough to be reproducible. More seriously, the headline results, hyperparameters, masking procedures, and memory measurements contradict one another across the paper. These problems prevent confidence in either the method or the empirical conclusions.

Critical strengths

Reducing the cost of preference alignment for multimodal models is a timely and practically meaningful objective. A method that retains model quality while reducing PPO training cost would be valuable, particularly for groups with limited computational resources. The paper also attempts to evaluate multiple model sizes and report accuracy, training time, FLOPs, and memory rather than considering task performance alone.

The authors provide ablations for cross-modal decoupling, adaptive sparsity, and progressive RL across multiple datasets and model sizes. This is the appropriate general structure for evaluating a framework containing several distinct components. Unfortunately, the value of these ablations is substantially reduced by inconsistencies between the main results and appendix tables.

Critical weaknesses

The paper does not clearly distinguish between masking weights, gradients, optimizer updates, activations, or structured model components. Equation 6 applies the mask, which suggests that parameters are removed during the forward pass. Elsewhere, inactive parameters are described as “frozen,” which would preserve them during the forward pass and only suppress their updates. These are fundamentally different algorithms. If only gradients are masked, most forward and backward computation remains dense, and the claimed FLOP reductions do not follow. If weights are removed, the authors must explain how input-dependent, unstructured parameter sparsity is converted into efficient GPU execution.

The importance score also depends on the gradient of every parameter. Computing these scores appears to require a dense backward pass before selecting the sparse subset, potentially eliminating the claimed savings. The paper does not explain whether gradients are computed densely, estimated from a separate batch, or reused across mask-update intervals. Nor does it describe how optimizer states are managed when the active subset changes.

Several mathematical terms are additionally problematic. The norm of a fixed-size hard Top K mask is constant and therefore cannot optimize mask selection. The gradient-truncation penalty appears to require second-order derivatives unless its gradient is detached; if detached, it provides no training signal. The KL divergence between deterministic Bernoulli masks is either zero or infinite, while the momentum update of two binary masks produces a non-binary mask. The paper does not provide a continuous relaxation, straight-through estimator, or thresholding procedure that resolves these issues.

The mask definition also changes across sections. The main method selects parameters using gradient magnitude, private-representation variance, and activation norms, whereas the appendix uses weight magnitude. Mask-update intervals are reported as both 50 and 10 steps. Consequently, it is unclear which algorithm was actually implemented.

Headline results differ dramatically between the main paper and appendix. For example, the reported Qwen-2-VL-2B MM-Vet score is 47.8 in the main table but 58.3 in the appendix. For Qwen-2-VL-7B on MMBench, the corresponding values are 81.8 and 59.8. The 7B VQAv2 result is variously reported as 81.2, 79.4, and 81.5. Differences of this magnitude cannot reasonably be attributed to random seeds. Similar inconsistencies occur for the 32B experiments.

The reward-stability motivation is also contradicted by the manuscript’s own values: AdaSparse-PRL is reported to have reward variance 0.28, compared with 0.26–0.27 for DPO and PPO, despite the text claiming lower variance. Hyperparameters likewise conflict: the initial KL coefficient is given as both 0.5 and 1.0.

Memory reporting is similarly unclear. The paper reports approximately 14.2 GB for the 7B model in one section, 18.2–20.3 GB in another, and approximately 36 GB or 48.6–51.2 GB for the 32B configuration in different places. The manuscript does not provide enough information about batch size, sequence length, quantization, checkpointing, ZeRO configuration, or offloading to reconcile these numbers.

Baseline fairness is also uncertain. The proposed method uses 4-bit loading, gradient checkpointing, ZeRO-3, CPU offloading, and sparse optimizer states, but it is not clear whether the same systems optimizations are applied to PPO, LoRA, QLoRA, and other baselines. Thus, the reported efficiency improvements may reflect unequal infrastructure settings rather than the proposed algorithm.

Finally, benchmark evaluation is underspecified. MM-Vet requires a clearly identified evaluator and scoring pipeline, while MMBench requires the split, language version, circular-evaluation protocol, and answer-extraction method. The paper instead reports generic “accuracy” without sufficient details. It also provides no confidence intervals or per-seed results for the main accuracy claims.

The theoretical analysis does not resolve these concerns. It reduces the method to a generic stochastic optimization argument and does not analyze hard TopK selection, changing masks, PPO clipping, learned reward-model error, or policy-generated data. The claimed improvement in the convergence rate relies on an unsupported assumption concerning variance reduction under adaptive TopK selection.

Overall, the paper addresses a worthwhile problem, but the current manuscript does not define a coherent, reproducible algorithm. The extensive numerical contradictions further prevent the reported results from being treated as reliable evidence. I therefore recommend strong reject.

Questions for authors

Clarification: What exactly is masked: forward-pass weights, gradients, optimizer updates, activations, or structured units? Please provide executable pseudocode for one complete training step, including mask construction and optimizer-state handling.

Clarification: How are per-parameter gradient importance scores computed without first performing a dense backward pass? Is the cost of scoring and global TopK selection included in all FLOP, memory, and wall-clock measurements?

Which reported results are correct? In particular, please reconcile the main and appendix results for the 2B, 7B, and 32B models, including the three different VQAv2 values reported for the 7B configuration.

Which mask rule was actually used: gradient/variance/activation-based selection or weight-magnitude selection? Was the mask updated every 10 or 50 steps?

Were all baselines evaluated with identical quantization, gradient checkpointing, ZeRO stage, CPU offloading, sequence length, batch size, rollout budget, and number of optimization steps?

What exact evaluation pipelines were used for MM-Vet, MMBench, and VQAv2, including evaluator models, dataset splits, prompt templates, decoding settings, and answer normalization?

Reviewer_RiCEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AdaSparse-PRL combines the decomposition of multimodal representations into shared and private components, input-dependent parameter masking, and a three-phase preference-alignment procedure that includes reward-model training, PPO with a decaying KL coefficient, and final mask alignment. The authors report higher benchmark scores and lower training costs on MM-Vet, MMBench, and VQAv2 using Qwen2-VL and Qwen2.5-VL. Appendix G.2 extends the evaluation to the Qwen3.5 model family.

Critical strengths
  1. The paper addresses resource-efficient multimodal preference tuning through an integrated pipeline for representation decomposition, parameter selection, and preference alignment
  2. The evaluation includes component ablations as well as multiple model scales and implementation frameworks.
Critical weaknesses
  1. The mask formulation and theoretical analysis need clarification.
  • Under the stated binary-mask definition, the KL term in Eq. (9) becomes infinite when a coordinate changes.
  • In Appendix C, the transition to Eq. (15) drops the squared-gradient term from the smoothness remainder.
  1. Inconsistencies in the reported results and insufficient efficiency analysis
  • The results in the ablation tables differ from Table 1 for 2B and 32B in MM-Vet and for all three model sizes in Table 3.
  • Section 3 also describes reward variances of 0.26–0.27 as higher than 0.28, and these values differ from Figure 1(c).
  • Appendix D does not explain how the cost of computing gradient-based parameter scores is included in the complexity analysis.
Questions for authors
  1. Which parameters are masked, and does masking affect the forward pass, parameter updates, or both?

  2. How is the KL term in Eq. (9) kept finite?

  3. What explains the differences between the scores in Table 1 and Tables 2–4?

Reviewer_KkzwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the inefficiency of MLLM fine-tuning, which it attributes to cross-modal entanglement, redundant parameter updates, and unstable preference alignment. It proposes AdaSparse-PRL, combining cross-modal representation decoupling, adaptive sparse parameter updates, and progressive preference optimization. Experiments on multimodal benchmarks report improved training efficiency while maintaining task accuracy.

Critical strengths
  1. The paper focuses on several key challenges in MLLM fine-tuning, namely cross-modal interference, redundant parameter updates, and preference alignment. It brings representation decoupling, sparse updates, and progressive optimization into one framework, offering an integrated approach to efficient multimodal adaptation. And Section 3 gives an analysis of their motivation.
  2. The evaluation considers training time and memory use alongside task performance, and includes component ablations and an optimized-framework comparison. This provides a broad assessment of the method's accuracy and efficiency.
Critical weaknesses
  1. The method reads more as a combination of strategies than as an in-depth investigation of the problems identified in Section 3. Some methodological details are unclear, and descriptions differ across sections, making the method difficult to follow.
  2. The reliability of the experimental results remains uncertain. In particular, the main paper and appendix report large, unexplained discrepancies for the same method and benchmarks. Without an explanation for these differences, it is difficult to determine which results reflect the method's actual performance and whether the reported gains are reliable.
Questions for authors
  1. Are the decoupling objective in Section 4.1 and the sparse task objective in Section 4.2 optimized jointly or in separate stages? If jointly, how are they combined; if separately, are the decoupling parameters frozen during sparse fine-tuning?
  2. What role do the shared and private representations from Section 4.1 play during sparse fine-tuning? How they are used in the forward pass?
  3. Do Table 1 and Appendix E use different checkpoints or evaluation settings? Table 1 reports 47.8 on 2B MM-Vet and 81.8 on 7B MMBench, whereas Appendix E reports 58.3 and 59.8, respectively. If the settings are the same, an explanation of these differences would be helpful.
  4. Appendix A.2 states that FLOPs include mask-generation costs and refers to gating networks. Could the authors clarify how the gradient-based score computation and Top-K selection are accounted for in the reported efficiency results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.