- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies model merging for multimodal large language models. The authors analyze the limitations of existing methods and propose CPCM, a training-free conformal prediction-calibrated curriculum merging framework. CPCM uses conformal prediction to assess each expert’s prediction reliability, which determines expert-level merging weights and a curriculum-based merging order. It then performs module-level weighted merging using a benchmark–module affinity matrix. The authors evaluate CPCM on multiple multimodal benchmarks and provide additional analyses to support its effectiveness.
Critical strengths
The paper proposes a simple, training-free method for merging multimodal large language models, making it practical to apply. Beyond this practical advantage, its use of conformal prediction to estimate expert reliability from predictive behavior offers a relatively underexplored perspective on model merging, supported by an insightful experimental analysis of module-level merging. I consider this combination of practical simplicity and methodological insight to be the paper’s main strength.
Critical weaknesses
- The method’s applicability is limited to multiple-choice tasks. The nonconformity score relies on a single correct answer, while parts of the method, such as normalization, depend on the number of answer options. These mechanisms are not directly defined for open-ended tasks such as captioning and free-form question answering, which are important applications of multimodal large language models. The paper does not discuss how it could be extended to these tasks.
- The experimental evidence is insufficient to substantiate the proposed design. The paper does not clearly explain how the benchmark–module affinity matrix is constructed or examine sensitivity to its choices. It is therefore unclear whether the reported effectiveness depends on extensive trial and error to identify a suitable matrix. Other hyperparameters, such as the lower and upper bounds on expert participation, are also insufficiently discussed. The ablation studies do not establish whether the proposed mechanisms outperform simpler alternatives. For example, comparisons could replace the weight-based curriculum order with a fixed or random order, or replace conformal prediction with average accuracy, entropy, or confidence as a measure of expert reliability. These two weaknesses leave me concerned about the effectiveness and generalizability of the proposed method for merging multimodal large language models.
Questions for authors
- Applicability to open-ended tasks. Can CPCM be applied to tasks such as image captioning and open-ended question answering? If so, how are its mechanisms adapted to these settings, and how does it perform on the corresponding benchmarks?
- Hyperparameter sensitivity. How is the benchmark–module affinity matrix constructed? How sensitive is CPCM’s performance to this matrix and other hyperparameters, such as the lower and upper bounds on expert participation? How are these choices determined, and does achieving good performance require extensive tuning?
- Ablations with simpler alternatives. How does performance change when individual mechanisms are removed or replaced with simpler alternatives? For example, what happens if the weight-based curriculum order is replaced with a fixed or random order, or if expert reliability is measured using average accuracy, entropy, or confidence instead of conformal prediction?