acceptodds
Withdrawn from ICLR 2027

CPCM: Conformal Prediction-calibrated Curriculum Model Merging for MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuning MLLMs on task-specific multimodal data has produced many experts with complementary capabilities. Model merging offers a practical way to integrate these experts into a unified MLLM without joint training or multi-model serving. However, existing merging methods mainly preserve task-specific knowledge through parameter-space heuristics, such as task-vector alignment, pruning, or reweighting, estimating expert capability contribution from weight-space operations rather than statistically calibrated prediction behavior. Moreover, reducing this calibrated signal to a single scalar weight for each expert task vector cannot distinguish the disentangled contributions of different modules. To address these issues, we introduce CPCM, a Conformal Prediction-calibrated Curriculum model Merging framework designed for MLLM expert merging. Specifically, at the expert level, we quantify expert prediction reliability via conformal prediction sets with statistical validity guarantees, deriving curriculum weights and ordering accordingly. At the module level, we further employ a benchmark-module affinity mechanism that projects benchmark-level reliability improvements into module-wise merging strengths for the visual encoder, multimodal projector, and language decoder. Experiments on multiple multimodal benchmarks show that our method surpasses strong task-vector merging baselines, while module-wise weight allocation further enhances the preservation of complementary expert capabilities.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Model Merging, Multimodal Large Language Model, Conformal Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k58POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies model merging for multimodal large language models. The authors analyze the limitations of existing methods and propose CPCM, a training-free conformal prediction-calibrated curriculum merging framework. CPCM uses conformal prediction to assess each expert’s prediction reliability, which determines expert-level merging weights and a curriculum-based merging order. It then performs module-level weighted merging using a benchmark–module affinity matrix. The authors evaluate CPCM on multiple multimodal benchmarks and provide additional analyses to support its effectiveness.

Critical strengths

The paper proposes a simple, training-free method for merging multimodal large language models, making it practical to apply. Beyond this practical advantage, its use of conformal prediction to estimate expert reliability from predictive behavior offers a relatively underexplored perspective on model merging, supported by an insightful experimental analysis of module-level merging. I consider this combination of practical simplicity and methodological insight to be the paper’s main strength.

Critical weaknesses
  1. The method’s applicability is limited to multiple-choice tasks. The nonconformity score relies on a single correct answer, while parts of the method, such as normalization, depend on the number of answer options. These mechanisms are not directly defined for open-ended tasks such as captioning and free-form question answering, which are important applications of multimodal large language models. The paper does not discuss how it could be extended to these tasks.
  2. The experimental evidence is insufficient to substantiate the proposed design. The paper does not clearly explain how the benchmark–module affinity matrix is constructed or examine sensitivity to its choices. It is therefore unclear whether the reported effectiveness depends on extensive trial and error to identify a suitable matrix. Other hyperparameters, such as the lower and upper bounds on expert participation, are also insufficiently discussed. The ablation studies do not establish whether the proposed mechanisms outperform simpler alternatives. For example, comparisons could replace the weight-based curriculum order with a fixed or random order, or replace conformal prediction with average accuracy, entropy, or confidence as a measure of expert reliability. These two weaknesses leave me concerned about the effectiveness and generalizability of the proposed method for merging multimodal large language models.
Questions for authors
  1. Applicability to open-ended tasks. Can CPCM be applied to tasks such as image captioning and open-ended question answering? If so, how are its mechanisms adapted to these settings, and how does it perform on the corresponding benchmarks?
  2. Hyperparameter sensitivity. How is the benchmark–module affinity matrix constructed? How sensitive is CPCM’s performance to this matrix and other hyperparameters, such as the lower and upper bounds on expert participation? How are these choices determined, and does achieving good performance require extensive tuning?
  3. Ablations with simpler alternatives. How does performance change when individual mechanisms are removed or replaced with simpler alternatives? For example, what happens if the weight-based curriculum order is replaced with a fixed or random order, or if expert reliability is measured using average accuracy, entropy, or confidence instead of conformal prediction?
Reviewer_YqP8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed CPCM, a Conformal Prediction-calibrated Curriculum model Merging framework designed for MLLM expert merging. The merging is conducted in two levels, namely the expert level and module level. Experiments merge four MLLM experts on Qwen2.5‑VL‑7B, evaluated across five multimodal benchmarks covering general VQA, document and spatial understanding, CPCM surpasses CMM,, Task arithmetic, TIES-Merging and DARE variants.

Critical strengths

The MLLM model merging is interesting, with existing methods performing on parameter-space signal. The proposed method performs expert merging.

The two-level merging is performed. The expert-level curriculum order and weights from CP reliability and further distributes strength across the heterogeneous MLLM sub-modules via benchmark-module affinity.

Experiment comparisons with existing merging baslines demonstrate the effectivenesses.

Critical weaknesses

The experiment setyp is limited, which only confines to Qwen2.5-VL-7B with four expert checkpoints. No cross-backbone validation, no large-scale expert-pool experiments. The generalization ability of CPCM can not be well jusitified.

Detailed experiments should be further justified. Fore example, the benchmark‑module affinity matrix is not learned end‑to‑end, which is interpreted / derived from benchmark characteristics. As such, how to extend to the new downstream task.

The nonconformity metric is based on the probability of the ground‑truth option, which only fits multiple‑choice problems. How to adapt this CP‑reliability pipeline for open‑ended generative multimodal tasks without special scoring heuristics?

Questions for authors

Please check the detailed information in the weaknesses part.

Reviewer_Fd4oOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CPCM uses prediction-set sizes to determine expert merging weights and order, then assigns module-specific coefficients through a benchmark–module affinity matrix. Experiments merge four Qwen2.5-VL-based experts and report improvements across five benchmarks.

Critical strengths

Prediction-based expert assessment is practically relevant, and the reported comparisons include both global and module-wise weighting.

Critical weaknesses
  • The conformal guarantee does not justify the expert reliability score. Thresholds calibrated on the base model are applied to different experts, which does not automatically preserve coverage. Smaller prediction sets can therefore represent confidently incorrect predictions.

  • The curriculum formulation is inconsistent. Fixed task-vector addition is order-independent, so the stated equations cannot produce a curriculum effect. Appendix A.5 instead describes changing the base after each step, implying a different algorithm.

  • Experimental validation is insufficiently controlled. Calibration–evaluation separation is unspecified, only CPCM receives labeled calibration data, and comparable accuracy/NLL-based weighting controls are missing. Module-wise allocation adds only 0.09 average points, without uncertainty estimates.

Questions for authors
  • What establishes expert coverage under base-model thresholds? Please report held-out coverage and prediction-set sizes.

  • What is the exact implemented merging equation, and does changing expert order affect the result?

  • Are calibration examples excluded from evaluation, and do gains persist against controls using the same labeled data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.