- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a specialize-before-fusion framework for video highlight detection and summarization: vision and audio each pass through independently parameterized bidirectional GRU branches, are conditioned on a text query, and are fused by a lightweight router weighting the branches at each time position. Overall, I found the execution reasonable and the ablations thorough, but the framing should be more carefully addressed. I will finalize my score after the authors' rebuttal.
Critical strengths
Broad and careful experiments. Four benchmarks, three seeds with standard deviations, baselines retrained on matched splits and features, and parameter counts and GFLOPs reported alongside accuracy.
The ablations are thorough. Branch capacity, routing, temporal layer type, clip length, feature extractors, prompt form, and cross-dataset transfer are all covered.
The representation analysis is valuable. UMAP and CKA show cross-modal branches separating near salient moments, and the authors connect same-modality redundancy to the diminishing returns of branch capacity.
Critical weaknesses
-
Most of the lead cannot be attributed to the proposed principle. Making TripleSumm's temporal module modality-specific gains only one to two points and still leaves it far below this method, so the gap comes from other choices, such as the temporal layer, an extra GRU prediction module, and the residual path. The principle's own effect is consistent but small, mostly one to two points. There is also no 2x2 experiment crossing shared or modality-specific branches with routing or no routing, so the two contributions cannot be separated. The title and abstract credit the principle for the results, so if most of the gain comes from elsewhere, the paper's own evidence does not back its main message.
-
The difference from prior work is not explained. Modeling each modality separately before fusion is common, for example in MulT [1], and the router is not compared against MoE-style routing [2] or attentional pooling. Together with the first point, I am not sure what the reader takes away beyond a well-built system, and this is the main reason for my score.
References
[1] Tsai et al. Multimodal transformer for unaligned multimodal language sequences (MulT). ACL 2019.
[2] Shazeer et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017.
Questions for authors
- Can you run the 2x2 experiment such as shared or modality-specific branches, with or without routing, and add the temporal layer, GRU prediction module, and residual path to TripleSumm, or remove them from your model? This would show how much of the lead belongs to the principle.
- How does specialize-before-fusion differ in substance from per-modality processing before fusion in MulT [1] and similar models?
- How does the router compare with MoE-style routing [2] or attentional pooling?
- Do routing weights vary over time across the test set like variance, entropy, and do they correlate with per-modality prediction quality or highlight labels? A single example and the masking case are not enough, and adaptive routing gains little over static routing.
- Does CKA on the shared-parameter model show the same separation near salient moments? Without it, the separation may simply mean that salient moments contain more distinct audio and video.
- The abstract claims lower inference cost, but this holds only for GFLOPs against TripleSumm, while the parameter count is about four times larger. Could you report parameter counts in the abstract and qualify the claim? Also, why does SG-DETR appear only in the unimodal comparison and not in the main tables?
References
[1] Tsai et al. Multimodal transformer for unaligned multimodal language sequences (MulT). ACL 2019.
[2] Shazeer et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017.
We thank the reviewer for the follow-up and are glad the new experiments addressed the concerns on attribution and motivation. We respect the reviewer’s view on novelty. We would only note that our main contribution is the controlled finding that, at matched capacity, sharing temporal parameters across modalities hurts (Table 4), which runs against what prior work (TripleSumm) reported and which we believe is useful beyond our specific model.