CPCM: Conformal Prediction-calibrated Curriculum Model Merging for MLLMs

Fine-tuning MLLMs on task-specific multimodal data has produced many experts with complementary capabilities. Model merging offers a practical way to integrate these experts into a unified MLLM without joint training or multi-model serving. However, existing merging methods mainly preserve task-specific knowledge through parameter-space heuristics, such as task-vector alignment, pruning, or reweighting, estimating expert capability contribution from weight-space operations rather than statistically calibrated prediction behavior. Moreover, reducing this calibrated signal to a single scalar weight for each expert task vector cannot distinguish the disentangled contributions of different modules. To address these issues, we introduce CPCM, a Conformal Prediction-calibrated Curriculum model Merging framework designed for MLLM expert merging. Specifically, at the expert level, we quantify expert prediction reliability via conformal prediction sets with statistical validity guarantees, deriving curriculum weights and ordering accordingly. At the module level, we further employ a benchmark-module affinity mechanism that projects benchmark-level reliability improvements into module-wise merging strengths for the visual encoder, multimodal projector, and language decoder. Experiments on multiple multimodal benchmarks show that our method surpasses strong task-vector merging baselines, while module-wise weight allocation further enhances the preservation of complementary expert capabilities.