acceptodds
Withdrawn from ICLR 2027

Dynamic Merging Mutually-Distilled Multi-View Teacher Network for RGB-Event based Representation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi‑modal fusion algorithms combining conventional frame‑based cameras and event cameras have recently attracted substantial research attention. Typically, existing approaches convert event streams into frames, voxels, or tensors, and then learn the required spatiotemporal information from these transformed event representations for fusion with conventional frame‑camera data. Obviously, such paradigms only exploit the limited representational capacity of event data. Moreover, standard training pipelines fail to fully leverage diverse model parameters obtained throughout the learning process. In this paper, we propose a novel mutual distillation method leveraging multi‑view teacher network weights to better integrate multiple event representations and achieve model‑parameter level fusion. Specifically, we employ multiple encoders to extract different types of event representations and perform mutual knowledge distillation at both the feature‑level and logit‑level in this process. Afterwards, we perform dynamic hierarchical model merge strategy on the model parameters from the first stage to yield a unified multi‑modal RGB‑Event representation model. Experimental results across multiple event-based datasets demonstrate that our proposed framework achieves significant performance improvements.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Model Merging, Multi-view Learning, RGB-Event, Event-based Vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UmUvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-view collaborative learning and dynamic hierarchical model merging framework for RGB-Event representation learning. First, the authors construct three representations from raw event streams, namely event frames, event streams, and event voxels, and establish three independent RGB-Event fusion branches. Logit-level and feature-level mutual distillation are then employed to facilitate knowledge transfer across branches, where feature-level distillation uses adaptive teacher weights based on prediction confidence and feature similarity.

Building upon this collaborative learning framework, the authors introduce Dynamic Hierarchical Model Merging (DHMM). The method first aggregates historical training checkpoints through learnable weighted averaging, and then merges model parameters from different branches using frequency-domain high-pass filtering and sparse expert extraction.

Experiments are conducted on two RGB-Event recognition datasets, PokerEvent and HARDVS, with ablation studies covering multi-view collaborative learning, model merging strategies, and relevant hyperparameters.

Critical strengths

Well-motivated collaborative learning across multiple event representations. Event frames, event streams, and event voxels preserve different forms of spatiotemporal information from event data. The authors employ three independent branches with bidirectional knowledge distillation at both the feature and logit levels to facilitate information exchange, which is consistent with the motivation of the paper. The method achieves good performance, supported by reasonably comprehensive ablation studies. In addition, Figure 4 reports a reduction in GPU memory usage from 5642 MB to 3442 MB after merging, demonstrating the potential value of the method for model deployment.

Critical weaknesses
  1. The frequency-domain model merging module is closely related to existing work, but the most direct comparison is missing. The frequency-aware merging module introduced in Section 3.4 applies Fourier transforms and high-pass filtering to task vectors to extract shared parameters, while using sparse experts to compensate for branch-specific information. This design is conceptually very close to FREE-Merging (Zheng & Wang, ICCV 2025), which is already cited by the authors. However, although Table 4 in Appendix B compares several model merging methods, FREE-Merging is not included. This comparison is particularly relevant given that Table 3 reports only a 0.45 percentage-point improvement of the full DHMM over CML alone. Since FREE-Merging was published in 2025, I would encourage the authors to include a direct comparison with this work and clarify the similarities and differences between the two methods.

  2. The construction of the final unified model and its inference procedure require further clarification. The paper emphasizes that model merging integrates three RGB-Event branches into a unified inference model. However, Eq. (13) in Section 3.4 defines branch-specific parameters θ_out,m rather than explicitly specifying a single set of final model parameters. Meanwhile, Appendix A.2 states that merging is applied only to the RGB-Event fusion and classification-head parameter groups. Therefore, a more detailed explanation is needed regarding how the three different event encoders are handled during inference and how the sparse experts from different branches contribute to the final predictions.

Questions for authors
  1. Differences from FREE-Merging and the independent contribution of DHMM. Have the authors considered directly applying FREE-Merging to the same CML-trained models and comparing the results with the proposed DHMM? Such an experiment would help clarify the additional value of dynamic historical checkpoint aggregation and multi-view hierarchical merging over existing frequency-domain model merging methods.

  2. Implementation of the final unified inference model. Could the authors further explain how the branch-specific parameters θ_out,m in Eq. (13) are combined to construct the final model? Does inference still require computing all three event representations and their corresponding encoding branches? How are the sparse experts selected or combined during inference?

Reviewer_KGfAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified RGB-event recognition framework that leverages three event representations through multi-level mutual distillation. It further introduces checkpoint interpolation and frequency-domain parameter merging to integrate knowledge across branches. The conducted experiments demonstrate the effectiveness of the proposed method.

Critical strengths
  • Exploring the complementary information among event frames, streams, and voxels is interesting for improving event-based recognition.

  • The collaborative learning strategy demonstrates clear benefits, achieving a significant performance improvement over the best independently trained branch.

  • The idea of consolidating knowledge from multiple branches into a more compact model is practically valuable, as it balances the benefits of collaborative learning with deployment efficiency.

Critical weaknesses
  • The frequency-domain merging strategy closely resembles FREE-Merging, yet this closely related method is not included in the comparisons. The authors should clarify the key differences and demonstrate the benefits of extracting sparse experts from low-frequency residuals.

  • The proposed adaptive designs lack comparisons with simpler alternatives. For example, historical checkpoint aggregation could be compared with uniform averaging, while the adaptive feature teacher could be compared with a fixed peer average. It is also important to ensure that different merging methods use the same source checkpoints for fair comparison.

  • Some reported improvements are rather marginal, particularly those brought by the historical merging stage.

  • Eq. (13) produces branch-specific parameters, which seems inconsistent with the claimed unified inference model. It would be better to clarify how the final model is constructed, which branches and encoders remain active, and how predictions are generated during inference.

  • Several implementation details require clarification. In particular, the number of historical checkpoints reported in Appendix A.2 differs from Table 5. The adaptive weighting, expert rescaling, and initialization of merged parameters should also be specified to improve reproducibility.

Questions for authors

Please see the weaknesses section.

Reviewer_w4jnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RGB-event recognition using three event representations: frames, discretized streams, and voxels. View-specific stems feed three RGB-event branches trained with classification, peer-logit distillation, adaptive feature distillation, and relation preservation. A subsequent merging stage learns convex combinations of historical checkpoints and applies Fourier-domain task-vector filtering with sparse residual experts. The authors report 58.73% Top-1 on PokerEvent and 53.28% Top-1 / 63.57% Top-5 on HARDVS. PokerEvent ablations attribute a 1.80 percentage-point gain to collaborative learning and a further 0.45 points to hierarchical merging. A before/after comparison also reports reduced GPU memory usage. The central question is whether heterogeneous views and optimization histories can be consolidated without losing complementary knowledge.

Critical strengths
  1. A useful combination of heterogeneous event views.The framework explicitly distinguishes event frames, temporally organized streams, and voxel tokens rather than treating events as a single homogeneous tensor. PokerEvent results improve from 56.48% for the strongest independently trained branch to 58.28% with collaborative learning (Table 3). This is the clearest evidence supporting the representation-learning contribution, subject to clarification of how the CML prediction is formed.
  2. Evaluation includes component and deployment analyses.The paper evaluates two recognition datasets and reports component, distillation, checkpoint-count, and merging-hyperparameter ablations (Tables 3 and 5-7), together with a before/after memory comparison (Fig. 4). These analyses make the proposal more assessable than a final-accuracy comparison alone. The reported memory reduction could be practically useful if tied to a precisely specified inference graph.
Critical weaknesses

1.The proposed consolidation is not specified sufficiently to establish what the final model is.Section 3.4 describes historical averaging within each branch, but subsequently introduces m in {f, s, v, hist}, leaving unclear whether hist is a fourth model or replaces each view checkpoint. Equation (13) produces view-indexed outputs rather than an unambiguous single predictor. Appendix A.2 states that only fusion and classification-head parameter groups are merged; it does not explain which stems and backbones remain, how their features reach the merged groups, or whether inference needs one or all event views. Thus, the claimed unified model and the memory comparison cannot be interpreted reliably. Reproduction also requires the common initialization for merged groups, the two-dimensional FFT reshape and mask convention, exact definitions of lambda_m and mu_m, and the construction/training of adaptive peer weights. The association of high-frequency parameter components with transferable knowledge is asserted rather than directly tested. This concern is about the central algorithm, not merely presentation polish (Sec. 3.3-3.5; App. A.2). 2. The evidence does not yet isolate a reliable advantage of the specific merging design.The gain over the strongest listed competitor is 0.46 percentage points on PokerEvent (58.73 vs. 58.27) and 0.07 on HARDVS (53.28 vs. 53.21); merging adds 0.45 points over CML on PokerEvent. No run-to-run variation or paired uncertainty is reported. Table 4 omits Free-Merging, the cited frequency-based method most directly relevant to this component, and does not establish whether competing mergers receive identical CML checkpoints, parameter groups, retained experts, and tuning budgets. A prediction ensemble and a controlled three-copy same-view mutual-learning baseline are also absent, limiting attribution to heterogeneous views rather than extra training capacity. Table 7 lacks logit-plus-feature distillation without relation preservation. Finally, A.2 specifies six historical checkpoints, whereas Table 5 reports the best result with five. These issues weaken claims that historical averaging, frequency filtering, and sparse experts provide stable, individually justified benefits (Tables 1-7; App. A.2-B).

Questions for authors

1.What exactly is executed at inference? Please give the final forward graph, its input views, retained stems/backbones, merged parameter groups, and rule for producing one prediction. Explain the view-indexed outputs in Eq. (13), whether hist is a separate source, and whether the 186.2M parameters / 224.2G FLOPs / 78.37 ms in Table 1 describe the same post-merge predictor as Fig. 4. What predictor produces the 58.28% CML-only result? 2. Can the merging implementation and checkpoint selection be made reproducible? Please specify the FFT reshape, mask and cutoff convention, lambda_m, mu_m, common initialization and treatment of normalization statistics. How are the five checkpoints associated with 58.73% selected, given the six checkpoints listed in A.2? Which split selects checkpoints and hyperparameters, and what optimizer/budget learns beta? A.2 already states that coefficient fitting uses the training split; please distinguish fitting from selection. 3. How stable are the reported improvements? Please provide matched-seed comparisons of the strongest single branch, CML, and CML+DHMM on both datasets, including variability and paired uncertainty where possible. Can the 0.07-point HARDVS advantage and the PokerEvent merging gain be distinguished from training variation? Clarify which external results were rerun under identical input and evaluation settings. 4. Does the proposed mechanism beat matched alternatives? Using identical CML model sources and inference budgets, how does it compare with prediction ensembling, uniform within-branch checkpoint averaging, and Free-Merging? An unfiltered merging control would help establish the value of frequency separation. A three-copy same-view control would help isolate heterogeneous-view complementarity from mutual-learning capacity. 5. How is the adaptive teacher implemented? Please define the architecture and normalization of a_m,n, the gradient paths through its feature/confidence inputs, and A(.) in Eq. (14). What is the result with logit-plus-feature distillation but no relation term, and how does the learned teacher compare with uniform peer weighting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.