- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TAC-Merge for consolidating task-specific multimodal experts into a single static model. Its multimodal influence mapping module characterizes how regional expert updates affect downstream representations and responses to image and instruction changes, and combines influence-graph curvature matching with agreement with expert predictions. Its coupled merge control module then jointly selects regional merging coefficients using a local quadratic model in a compact, response-informed subspace.
Critical strengths
- The topic of merging task-specific multimodal models into a single model is interesting and important.
- The paper is generally well organized and easy to follow.
Critical weaknesses
- The main experiments use Qwen3.5-4B and Qwen3-VL-8B-Instruct, which limits direct comparison with published SOTA results obtained under earlier backbone and expert configurations. Moreover, Expert Merging [1] is cited but not included in the comparisons, despite also learning merging coefficients through expert alignment on unlabeled calibration data.
- Since the experiments use newly trained LoRA experts on Qwen3.5/3 backbones, it is difficult to assess the reproducibility and fairness of the comparisons with insufficient baseline implementation details and unknown hyperparameter selection procedures.
- LOT Merging [2] seems particularly relevant to the paper’s motivation concerning changes in intermediate representations during merging, but this work is not discussed.
- The evaluation is restricted to two Qwen-family backbones, leaving generalization to other model families unverified.
[1] Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking. ICLR, 2026.
[2] Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration. NeurIPS, 2025.
Questions for authors
- Can TAC-Merge be adapted to text-only LLM merging, a setting explored by RegMean++/Chain of Merges (CoM)? If so, how would it compare with these methods under a shared evaluation protocol?
- Could the authors visualize and analyze the final region-by-expert coefficient matrix, preferably showing deviations from the uniform initialization? In particular, how do the selected expert weights differ across the visual and language regions? Such an analysis would help clarify the coordination patterns produced by the controller beyond the performance improvements.
6 comments
To W2
Thank you for requesting these details. The baseline comparisons use the same expert checkpoints and evaluation settings within each backbone. Calibration-based methods share 32 unlabeled inputs per task. Evaluation uses greedy decoding with a 64-token limit, BF16 for Qwen and FP16 for LLaVA. All hyperparameter searches use the same validation criterion, minimizing KL divergence to task-expert predictions on separate unlabeled validation inputs from seen tasks, with equal weight assigned to each task. Parameter selection is completed before test evaluation.
The shared merge-scale grid is . Task Arithmetic scales the summed expert updates and searches the scale over , selecting 0.1. TIES fixes its retained fraction at 0.2 and resolves sign conflicts before aggregation; its merge scale is selected from , giving 1.0. DC-Merge retains its spectral smoothing and subspace-alignment procedure with fixed settings. CoM follows its sequential merging procedure with an off-diagonal Gram coefficient of 0.95. RegMean++ searches the off-diagonal coefficient over , selecting 0.1. LOT-Merge collects features from the original experts and solves for all 224 adapted projections. The projector follows the same fixed averaging policy as the other methods. RobustMerge fixes the retained fraction and attention ratio at 0.2 and searches the fusion weight over , selecting 0.875 for Qwen and 2.0 for LLaVA. OptMerge retains its task-vector optimization procedure and searches the final merge scale over , selecting 1.0. Expert Merging learns matrix-level coefficients through expert alignment on the shared unlabeled calibration inputs, incorporating the provided visual and language LoRA updates.
To W3
Thank you for highlighting LOT Merging. This work minimizes feature drift through layer-wise task-vector reconstruction, formulated as quadratic objectives with closed-form solutions. It is directly relevant to changes in intermediate representations during merging. We will explicitly discuss this connection in the revised related work.
The distinction concerns what is matched and how updates are coordinated. LOT reconstructs local expert features. TAC-Merge measures how regional expert updates affect downstream multimodal responses and uses the resulting influence structure for matching. Its controller then selects coefficients jointly across visual and language regions, accounting for interactions between regional updates.
We additionally implemented a LoRA adaptation of LOT on the public LLaVA-v1.5-7B experts released by RobustMerge, using the same 32 unlabeled calibration inputs per task as TAC-Merge. The adaptation collects features from the original experts and solves for all 224 adapted projections. The projector follows the same fixed averaging policy as the other methods. A projection-level comparison with an independent SVD solution produced a relative error of , and the exported model passed forward-generation checks.
As reported in our response to W1, LOT-Merge obtains average scores of 57.79 on seen tasks and 37.26 on unseen tasks. TAC-Merge improves these scores by 4.65 and 1.70 points, respectively. This comparison supports the benefit of joint influence coordination relative to local feature reconstruction in the shared LoRA merging setting.
To W1
Thank you for this comment. We have added LLaVA-v1.5-7B experiments using the public experts from RobustMerge to complement the Qwen results. The comparison includes Expert Merging and LOT-Merge. All methods share the expert checkpoints and evaluation settings. Calibration-based methods use the same 32 unlabeled inputs per task.
TAC-Merge achieves the highest average score on both seen and unseen tasks. Compared with Expert Merging, it improves the average score by 3.93 points on seen tasks and 1.55 points on unseen tasks.