- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a multi-view collaborative learning and dynamic hierarchical model merging framework for RGB-Event representation learning. First, the authors construct three representations from raw event streams, namely event frames, event streams, and event voxels, and establish three independent RGB-Event fusion branches. Logit-level and feature-level mutual distillation are then employed to facilitate knowledge transfer across branches, where feature-level distillation uses adaptive teacher weights based on prediction confidence and feature similarity.
Building upon this collaborative learning framework, the authors introduce Dynamic Hierarchical Model Merging (DHMM). The method first aggregates historical training checkpoints through learnable weighted averaging, and then merges model parameters from different branches using frequency-domain high-pass filtering and sparse expert extraction.
Experiments are conducted on two RGB-Event recognition datasets, PokerEvent and HARDVS, with ablation studies covering multi-view collaborative learning, model merging strategies, and relevant hyperparameters.
Critical strengths
Well-motivated collaborative learning across multiple event representations. Event frames, event streams, and event voxels preserve different forms of spatiotemporal information from event data. The authors employ three independent branches with bidirectional knowledge distillation at both the feature and logit levels to facilitate information exchange, which is consistent with the motivation of the paper. The method achieves good performance, supported by reasonably comprehensive ablation studies. In addition, Figure 4 reports a reduction in GPU memory usage from 5642 MB to 3442 MB after merging, demonstrating the potential value of the method for model deployment.
Critical weaknesses
-
The frequency-domain model merging module is closely related to existing work, but the most direct comparison is missing. The frequency-aware merging module introduced in Section 3.4 applies Fourier transforms and high-pass filtering to task vectors to extract shared parameters, while using sparse experts to compensate for branch-specific information. This design is conceptually very close to FREE-Merging (Zheng & Wang, ICCV 2025), which is already cited by the authors. However, although Table 4 in Appendix B compares several model merging methods, FREE-Merging is not included. This comparison is particularly relevant given that Table 3 reports only a 0.45 percentage-point improvement of the full DHMM over CML alone. Since FREE-Merging was published in 2025, I would encourage the authors to include a direct comparison with this work and clarify the similarities and differences between the two methods.
-
The construction of the final unified model and its inference procedure require further clarification. The paper emphasizes that model merging integrates three RGB-Event branches into a unified inference model. However, Eq. (13) in Section 3.4 defines branch-specific parameters θ_out,m rather than explicitly specifying a single set of final model parameters. Meanwhile, Appendix A.2 states that merging is applied only to the RGB-Event fusion and classification-head parameter groups. Therefore, a more detailed explanation is needed regarding how the three different event encoders are handled during inference and how the sparse experts from different branches contribute to the final predictions.
Questions for authors
-
Differences from FREE-Merging and the independent contribution of DHMM. Have the authors considered directly applying FREE-Merging to the same CML-trained models and comparing the results with the proposed DHMM? Such an experiment would help clarify the additional value of dynamic historical checkpoint aggregation and multi-view hierarchical merging over existing frequency-domain model merging methods.
-
Implementation of the final unified inference model. Could the authors further explain how the branch-specific parameters θ_out,m in Eq. (13) are combined to construct the final model? Does inference still require computing all three event representations and their corresponding encoding branches? How are the sparse experts selected or combined during inference?