Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging

Multimodal model merging consolidates task experts into a single model while preserving their complementary capabilities. Most existing approaches follow the layer-wise design of unimodal merging and combine the parameter changes learned by experts within each layer. However, merging one layer alters downstream representations and affects how visual and textual information interact. These dependencies make it difficult to trace the influence of individual experts across depth and jointly select merging weights. To tackle these challenges, we propose TAC-Merge, which contains two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM builds graphs to trace expert influence across depth and distinguish responses to image and instruction changes. The shared objective combines Ricci curvature matching on these graphs with prediction agreement to retain expert capabilities. CMC then uses the measured responses to identify a compact subspace for optimizing this objective. Within this subspace, CMC fits a local quadratic model of weight interactions and jointly selects regional weights to reduce interference. Experiments across diverse multimodal tasks demonstrate the effectiveness of TAC-Merge and its generalization to unseen tasks.