acceptodds
Under review as a conference paper at ICLR 2027

Specialize Before Fusion: Modality-Specific Routing for Multimodal Video Highlight Detection

Abstract

Highlight detection and video summarization require identifying salient moments from heterogeneous videos where visual and acoustic evidence can vary over time. Existing multimodal methods mainly focus on how modalities are fused, often before adequately modeling their distinct temporal characteristics. We introduce a specialize-before-fusion framework that first learns independently parameterized visual and audio temporal representations and then adaptively combines them using time-varying routing weights, optionally conditioned on a semantic query. Across four benchmarks spanning long-form sports broadcasts and web video, our method achieves state-of-the-art performance, improving over the strongest baseline by up to 6.9 mAP on long-form sports highlight detection while incurring lower inference cost. Ablations consistently favor modality-specific over shared temporal modeling and adaptive over fixed fusion, and representation analyses show visual and audio branches become more distinct specifically near salient moments. These results support specializing heterogeneous modalities before multimodal integration.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video highlight detection, Video summarization, Multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ikPmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a specialize-before-fusion framework for video highlight detection and summarization: vision and audio each pass through independently parameterized bidirectional GRU branches, are conditioned on a text query, and are fused by a lightweight router weighting the branches at each time position. Overall, I found the execution reasonable and the ablations thorough, but the framing should be more carefully addressed. I will finalize my score after the authors' rebuttal.

Critical strengths

Broad and careful experiments. Four benchmarks, three seeds with standard deviations, baselines retrained on matched splits and features, and parameter counts and GFLOPs reported alongside accuracy.

The ablations are thorough. Branch capacity, routing, temporal layer type, clip length, feature extractors, prompt form, and cross-dataset transfer are all covered.

The representation analysis is valuable. UMAP and CKA show cross-modal branches separating near salient moments, and the authors connect same-modality redundancy to the diminishing returns of branch capacity.

Critical weaknesses
  1. Most of the lead cannot be attributed to the proposed principle. Making TripleSumm's temporal module modality-specific gains only one to two points and still leaves it far below this method, so the gap comes from other choices, such as the temporal layer, an extra GRU prediction module, and the residual path. The principle's own effect is consistent but small, mostly one to two points. There is also no 2x2 experiment crossing shared or modality-specific branches with routing or no routing, so the two contributions cannot be separated. The title and abstract credit the principle for the results, so if most of the gain comes from elsewhere, the paper's own evidence does not back its main message.

  2. The difference from prior work is not explained. Modeling each modality separately before fusion is common, for example in MulT [1], and the router is not compared against MoE-style routing [2] or attentional pooling. Together with the first point, I am not sure what the reader takes away beyond a well-built system, and this is the main reason for my score.

References

[1] Tsai et al. Multimodal transformer for unaligned multimodal language sequences (MulT). ACL 2019.

[2] Shazeer et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017.

Questions for authors
  1. Can you run the 2x2 experiment such as shared or modality-specific branches, with or without routing, and add the temporal layer, GRU prediction module, and residual path to TripleSumm, or remove them from your model? This would show how much of the lead belongs to the principle.
  2. How does specialize-before-fusion differ in substance from per-modality processing before fusion in MulT [1] and similar models?
  3. How does the router compare with MoE-style routing [2] or attentional pooling?
  4. Do routing weights vary over time across the test set like variance, entropy, and do they correlate with per-modality prediction quality or highlight labels? A single example and the masking case are not enough, and adaptive routing gains little over static routing.
  5. Does CKA on the shared-parameter model show the same separation near salient moments? Without it, the separation may simply mean that salient moments contain more distinct audio and video.
  6. The abstract claims lower inference cost, but this holds only for GFLOPs against TripleSumm, while the parameter count is about four times larger. Could you report parameter counts in the abstract and qualify the claim? Also, why does SG-DETR appear only in the unimodal comparison and not in the main tables?

References

[1] Tsai et al. Multimodal transformer for unaligned multimodal language sequences (MulT). ACL 2019.

[2] Shazeer et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017.

Reviewer_SGWhOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Paper is about video highlight detection. The proposal is to have specialized parts of a network for each modality (in this case audio and video) with different specialized parameters. Then to have a routing network that choose how much of which modality to let through for highlight detection.

Critical strengths

Paper takes an approach that in some ways may be against the trend (specializing rather than all modalities being processed the same way).

Ablation is rather thorough

Results and done over multiple benchmarks

Critical weaknesses

Not entirely sure the method described is as fundamentally novel. It seems more a juxtaposition of ideas into a new method.

Personally I was more curious to know how the router behaved. A deeper analysis of the router's behaviour for specialist modality networks vs a general one would potentially be educational for me.

Questions for authors

Why GRU?

Is there any effort to train the specialist networks with some unsupervised training to see if they can become general parts reusable over different problems.

I am not requiring this, but still documenting my observations :- I would be keen to know whether the modality encoders and the router learning is transferable across datasets and benchmarks. Most papers don't do cross-dataset reporting, but it can be really valuable as an insight into the utility of the proposed method for real-life scenarios.

Reviewer_5DcEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a specialize-before-fusion principle for multimodal highlight detection and video summarization. The idea is that the temporal representations of the visual and audio streams should be learned separately before the two are fused. Each modality gets its own BiGRU branches, with no parameter sharing across modalities. The branch outputs are conditioned on a text query through cross-attention and then combined by per-timestep softmax routing. The method outperforms prior work on both long-form sports highlight benchmarks and web video summarization benchmarks, with the largest gains on long-form sports. The paper also includes ablations on parameter sharing, routing strategy, number of branches, missing modalities, and representation similarity.

Critical strengths

S1. The paper shows large and consistent improvements on long-form sports highlight detection. Baselines were retrained with the same data splits and input features, and results are averaged over multiple seeds, so the comparison is reasonably fair. Strong long-form sports HD baselines that make proper use of audio are still lacking, so these results have practical value on their own.

S2. The paper includes a controlled experiment that directly targets the core hypothesis. With the parameter count held fixed, replacing shared branches with modality-specific ones consistently improves mAP. Removing parameter sharing from TripleSumm also improves its performance on the sports benchmarks. In other words, a design that earlier work reported as benefiting from sharing does not carry over to the sports domain. I find this the most convincing part of the paper.

Critical weaknesses

W1. The research direction seems meaningful, but the methodological novelty is somewhat limited. Modality-specific encoders followed by per-timestep softmax gating is not very different from soft MoE or existing multimodal gating approaches. On the sports datasets, the same prompt is used for every video, so the text query conditioning effectively reduces to a set of fixed learnable tokens. On SV-Highlights, the difference between using and not using a prompt is within one standard deviation.

W2. Neither the motivation nor the experiments sufficiently support the core claim. TripleSumm already handled the idea that modality importance changes over time, through adaptive fusion. What this paper newly claims is that the two modalities have different temporal structures and therefore need separate parameters, yet there is no analysis of how often such cases actually occur in the datasets. Figure 1 shows the routing output of the proposed model itself, so it is hard to treat it as evidence that the problem exists. On the experimental side, specialization contributes only around 1 to 2 mAP in the parameter-matched comparison. In contrast, replacing the GRU temporal layer with attention or an MLP drops performance below TripleSumm. This suggests that much of the headline gain likely comes from recurrent temporal modeling or differences in the training setup, rather than from specializing before fusion.

Questions for authors

Q1. Could the authors break down the performance gap with TripleSumm by component? For example, it would be useful to see TripleSumm with the same GRU temporal model and weighted BCE loss, or the proposed model with only the temporal model replaced by attention and everything else unchanged. It would also help to clarify whether the Attention variant in Table 11 replaces only the branches or also the GRU after fusion.

Q2. Does the No Prompt setting in the sports experiments remove the cross-attention module entirely? If so, the parameter count differs between the two settings. A comparison that replaces the fixed prompt with the same number of learnable tokens would show whether query conditioning actually uses semantic information.

Q3. Is there any data-level analysis that supports the motivation? For example, the authors could report how time offsets between audio reactions and visual events are distributed within highlight segments, what fraction of highlights can be detected from only one modality, and whether the gains over the shared model concentrate in such segments. Such an analysis would make it much clearer why specialization is needed.

Comments on the paper

1 comment

We thank the reviewer for the follow-up and are glad the new experiments addressed the concerns on attribution and motivation. We respect the reviewer’s view on novelty. We would only note that our main contribution is the controlled finding that, at matched capacity, sharing temporal parameters across modalities hurts (Table 4), which runs against what prior work (TripleSumm) reported and which we believe is useful beyond our specific model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.