- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies hallucination in multimodal model merging. On Qwen2.5-VL-3B/7B, it merges four LoRA experts (Chart, Geometry, OCR, VQA) with eight merging methods. Every merged checkpoint shows a higher AMBER hallucination rate than the average of its experts, which the authors call a "merging tax". To reduce it, they propose MergeHEIR. It applies mDPO-style preference optimization to the merged model. To protect the inherited expertise, it builds layer-wise null spaces from expert-task calibration activations and periodically projects the accumulated update onto them. The paper gives theoretical results on this projection. Experiments cover three backbones and eight merging methods, for 24 merged vs. MergeHEIR comparisons.
Critical strengths
-
The problem is new and useful. To my knowledge, this is the first systematic study showing that model merging can increase multimodal hallucination. The trend is consistent across 16 merged checkpoints. This matters for anyone who deploys merged MLLMs, so I count it as the main reason to consider acceptance.
-
The evaluation is broad. It covers eight merging methods, three backbones, four hallucination benchmarks, and seven expert-task benchmarks. Table 9 also shows that the projection can be added to other preference methods. This breadth makes the hallucination reduction itself convincing, even though the retention claim needs more support (see Weaknesses).
Critical weaknesses
-
The evidence for the null-space constraint is limited. The main comparisons are merged vs. merged+DPO, which mostly show that DPO reduces hallucination. The key comparison against unconstrained adaptation covers only three settings, with small gaps and no variance. Simple retention baselines are also missing. Since better retention is the method's main claim, this weighs most on my rating.
-
The merging tax is not well isolated. Without the base model, individual experts, and response-length controls, it is unclear whether the increase comes from merging itself. The proposed mechanism is also not tested. This matters because the merging tax is the paper's core motivation.
Questions for authors
-
The comparison with unconstrained adaptation (i.e., the same training without null-space projection) is only shown for one merging method per model. Could you run it on more merging methods, with multiple random seeds? It would also help to compare with a simple way to protect expert skills, for example adding a penalty that keeps the weights close to the merged model.
-
The merging tax is measured against the average of the experts. Could you also report the hallucination rate of the original base model and of each expert? Please also report the average response length, since shorter answers usually contain fewer hallucinations. This would show whether the increase really comes from merging.
Overall, I like this paper. The problem is new and interesting, and the paper is clearly written. The method itself is fairly simple, but I think this is fine if the experiments support the main claims. Right now, the two points above are not fully addressed. If the authors address them, I will keep my positive score. If not, I may lower it.
Suggestion for future work (not affecting my score): direct evidence for the proposed mechanism, such as an analysis of attention to visual tokens.