acceptodds
Withdrawn from ICLR 2027

UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

RGB-X tracking leverages complementary modalities, such as thermal infrared, event, or depth, to improve robustness under challenging conditions. Existing general-purpose RGB-X trackers typically follow two paradigms: independent task-specific training and joint training across modality combinations. However, the former suffers from knowledge isolation and model redundancy, while the latter introduces optimization interference across heterogeneous tasks. In this work, we present UniMergeTrack, a new model merging paradigm that preserves task-specific optimization while enabling effective cross-task knowledge integration. UniMergeTrack first trains modality-specific trackers independently and then performs hierarchical model merging. Specifically, transferable backbones are merged across RGB-X tasks via high-pass-filtered merging, and complemented by lightweight task-specific experts that preserve modality-specific cues. Tracking heads are merged within each task using optimal transport-guided channel alignment to mitigate hidden channel mismatch. A subsequent distribution alignment fine-tuning stage further improves compatibility between the unified backbone and task-specific heads. Extensive experiments on the LasHeR, RGBT234, VisEvent, VOT-RGBD22, COESOT, and CDTB benchmarks demonstrate that UniMergeTrack achieves state-of-the-art or competitive performance across RGB-Thermal, RGB-Event, and RGB-Depth tracking tasks, while exhibiting strong generalization to unseen datasets. The source code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RGB-X Tracking, Model Merging, Multimodal Learning, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UzjeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UniMergeTrack, a model-merging framework for unified RGB-Thermal, RGB-Depth, and RGB-Event visual object tracking. Starting from independently trained task-specific trackers, it merges their backbones using high-pass-filtered task vectors while retaining sparse modality-specific experts. Within each task, it combines tracking heads from different training checkpoints through optimal-transport-based channel alignment and parameter averaging. A final fine-tuning stage adapts the heads to the merged backbone while keeping the backbone frozen.

The main contribution is this hierarchical strategy of cross-task backbone merging and intra-task head merging. The paper evaluates the framework on multiple RGB-X tracking benchmarks, comparing it with task-specific trackers, unified trackers, and existing model-merging methods. It reports improved tracking performance and reduced storage relative to maintaining separate models, supported by component ablations and comparisons of merging and channel-alignment strategies.

Critical strengths
  1. Practical consolidation of modality-specific trackers. The framework combines RGB-Thermal, RGB-Depth, and RGB-Event trackers into a shared backbone with lightweight task-specific components. The reported storage reduction—from three 340 MB models to approximately 379 MB—while improving over independently trained trackers demonstrates a useful accuracy–storage trade-off. This gives the approach practical value beyond improvements in tracking scores alone.

  2. Broad empirical evaluation with targeted ablations. The paper evaluates multiple RGB-X tracking benchmarks and compares against both tracking and model-merging baselines. It also examines alternative backbone/head merging strategies and channel-alignment methods, rather than only removing individual modules. The reported improvements over FREE-Merging, together with these controlled comparisons, provide empirical support for the proposed tracking-specific integration, even though its individual techniques largely build on prior work.

Critical weaknesses
  1. Limited methodological novelty beyond existing model-merging techniques. The backbone-merging pipeline closely follows FREE-Merging, including frequency filtering, sparse task-specific experts, and rescaling. Likewise, aligning neurons with optimal transport before averaging parameters was established by Model Fusion via Optimal Transport (NeurIPS 2020). The main remaining contribution is therefore the particular combination of cross-task backbone merging and intra-task head merging for RGB-X tracking. Although this combination produces improvements, the paper does not sufficiently establish a new methodological insight beyond adapting and integrating existing techniques. This limits the significance of its claimed contribution.

  2. Insufficient evidence that the gains require the proposed pipeline under matched training budgets. Performance depends strongly on post-merging fine-tuning: without it, AUC falls from 61.8 to 57.7 on LasHeR and from 64.1 to 60.9 on VisEvent, below the independent-training baselines of 58.8 and 62.2. This does not invalidate merging, but makes budget-matched adaptation controls particularly important. The paper does not clearly establish whether the independent/joint-training baselines receive comparable additional training or whether all merging baselines use identical post-merging adaptation. Despite the component ablations, it remains unclear whether the hierarchical pipeline offers a meaningful advantage over simpler merging or checkpoint-averaging approaches with equally tuned heads. This weakens confidence in attributing the reported gains to the proposed design.

Questions for authors
  1. The main merging techniques in Stage 2—including frequency filtering, sparse expert extraction, and OT-based channel alignment—have precedents in existing work. Across the three stages, what do the authors consider the single most important technical innovation? Authors should clarify how it differs from prior methods and whether the contribution lies primarily in a new algorithm, the specific combination of existing techniques, or their adaptation to RGB-X tracking.
  2. The pipeline appears to introduce several corrective steps: filtering addresses interference from cross-task merging, sparse experts preserve task-specific information, and subsequent head adaptation addresses incompatibility with the merged backbone. This sequence is plausible, but what evidence shows that the full combination is necessary, rather than a simpler design avoiding some of these intermediate problems? The component ablations and alignment comparisons demonstrate improvements within the proposed configuration; could the authors explain which results establish the advantage of the overall design over simpler alternatives, and which specific failure modes make its additional steps necessary?
  3. Table 3 shows that, without post-merging fine-tuning, the merged model performs worse than the independent-training baseline on both LasHeR and VisEvent. Since the final stage uses the original tracking objective to adapt the heads, what evidence distinguishes the benefit of hierarchical merging from that of subsequent adaptation? In particular, do the merging baselines in Table 5 receive the same post-merging fine-tuning, and do the independent- and joint-training baselines receive comparable training budgets? Clarifying these conditions would help determine whether the reported gains specifically support the proposed merging strategy.
Reviewer_3MoMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniMergeTrack, which achieves unified RGB-X tracking through cross-task backbone merging, sparse modality experts, and optimal transport-based merging of intra-task tracking heads, followed by fine-tuning. The method is evaluated on six benchmarks, and ablation studies and strategy comparisons demonstrate competitive tracking performance, cross-dataset transferability, and storage advantages.

Critical strengths
  1. The design is sound. Cross-task backbone merging and intra-task tracking-head merging jointly balance sharing and specialization, and ablation studies support this design.
  2. It has practical value. The method achieves competitive performance on six benchmarks while substantially reducing the storage required to keep three separate models.
Critical weaknesses
  1. Limited novelty. High-pass merging, sparse experts, and optimal transport (OT) alignment all have precedents. The main contribution of this paper is a hierarchical combination tailored to the tracking task, with limited methodological breakthrough.
  2. Insufficient attribution of gains. The lack of comparisons under an equal training budget and with additional fine-tuning controls makes it difficult to disentangle the contributions of model merging and continued training, thereby weakening the core argument.
Questions for authors

Do the gains persist with matched training budgets? The full method appears to use 30 epochs of task-specific training plus 30 epochs of alignment fine-tuning, whereas the independent baseline uses only 30 epochs. Please compare against:

Independent training followed by 30 additional epochs of head-only fine-tuning with the backbone frozen.

Independent training for 60 epochs.

Which “other lightweight parameters” are trainable in the third stage?

If these baselines approach 61.8/77.2 on LasHeR and 64.1/81.5 on VisEvent (AUC/PR), the claimed advantage of model merging would be substantially weakened. A persistent gain of 2–3 percentage points would substantially strengthen my assessment.

Suggestions for future work: Evaluate robustness to missing or degraded modalities and routing errors. Explore routing without known modality labels and dynamic expert fusion to strengthen the framework’s generality.

Reviewer_wD2AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniMergeTrack trains separate RGB-T, RGB-E and RGB-D trackers, then merges their backbones using high-pass-filtered task vectors and sparse modality experts. Tracking heads are merged within each task after OT channel alignment, followed by head fine-tuning. Experiments on six benchmarks show competitive performance with lower storage than three separate models.

Critical strengths
  1. The evaluation covers three tracking modalities and transfer to COESOT and CDTB. Reported storage drops from 1,020 MB for separate models to 379 MB, which is useful for deployment.
  2. Tables 6-7 support merging backbones across tasks while merging heads within each task, and show a benefit from OT alignment.
Critical weaknesses
  1. The methodological novelty is limited. High-pass filtering and sparse experts were already introduced in FREE-Merging; the main addition is treating the backbone and tracking heads differently. Tables 6-7 support this design, but the explanation based on channel mismatch lacks direct evidence. The two heads come from epochs 24 and 30 of the same training run, yet the paper does not measure how their channel correspondence changes. The OT ablation shows a benefit, but does not establish that correcting channel mismatch explains the gain.
  2. The expert composition is ambiguous. Eqs. (9)-(10) define the expert as a sparse parameter update, while Eq. (14) uses it to replace the backbone weights directly. The authors should clarify whether the implementation adds or replaces weights and give consistent equations. The value of gamma used for expert extraction is also missing. These details matter for reproducing the method.
Questions for authors
  1. Do the independent, joint-training and merging baselines use matched training budgets and post-training fine-tuning?
  2. How much do channel correspondences change between the epoch-24 and epoch-30 heads? Can you provide alignment statistics to support the claimed channel mismatch?
  3. Is the sparse expert added to the merged weights or used to replace them in Eq. (14)? Please reconcile this with Eqs. (9)-(10) and specify the value of gamma.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.