acceptodds
Under review as a conference paper at ICLR 2027

Granularity-Aligned Text-to-Video Retrieval with Semantic Conduits

Abstract

Text-to-video retrieval (T2VR) aims to align textual descriptions with video content, yet the core challenge lies in the inherent asymmetry of information density between the two modalities. Existing approaches often overlook this asymmetry in feature construction, either compressing videos into global representations or relying on coarse frame-level features, which conflate distinct visual concepts and yield a granularity mismatch with token-level textual features. To address this, we propose a granularity-aligned retrieval paradigm (GAR) that organizes video contents into semantic units better matched to fine-grained textual features. Drawing on the patch-level visual granularity that underpins fine-grained image-text alignment, we instantiate this paradigm with a hierarchical, joint spatial-temporal video representation built upon Semantic Conduits. Appearance-similar patches are first coalesced within each frame into semantic prototypes and then associated across frames. Soft patch-to-conduit assignments aggregate temporally coherent visual evidence into pooled conduit tokens, with a Transformer contextualizes by modeling relations among semantic components. MeanMaxSim aligns each text token with its most similar conduit and averages the token-wise scores, yielding fine-grained cross-modal correspondence while permitting video representations to be precomputed. Experiments on four benchmarks demonstrate strong retrieval performance. With a ViT-B/32 backbone, our method achieves R@1 scores of 48.4% on MSR-VTT, 48.1% on MSVD, 48.8% on DiDeMo, and 46.0% on ActivityNet. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Video Retrieval, Video-Language Representation Learning, Fine-Grained Cross-Modal Alignment, Spatiotemporal Representation Learning, Late Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YZ5fOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Granularity-Aligned Retrieval (GAR), which first groups video patches into intra-frame semantic regions and then associates related regions across frames into Semantic Conduits, producing visual units with a granularity closer to text tokens. A Temporal Transformer contextualizes these conduits, and MeanMaxSim performs fine-grained token-to-conduit matching, achieving strong text-to-video retrieval performance with lower retrieval-stage storage and matching costs.

Critical strengths
  1. The paper is well-written and easy to understand.
  2. The experiments show the effectiveness of the proposed method.
Critical weaknesses
  1. The paper establishes spatiotemporal associations by partitioning regions based on patch similarity before matching them with the text. Why not use a segmentation method like SAM3 instead? That would likely be more accurate and computationally less expensive.
  2. There are already many existing fine-grained matching methods such as EagleNet; the paper should compare its approach against these fine-grained retrieval methods to better highlight its contributions.
  3. The current method is mainly object-oriented, as Semantic Conduits are constructed by associating appearance-similar regions across frames, while explicit modeling of fine-grained temporal dynamics is limited. It would therefore be valuable to evaluate the method on temporally fine-grained action benchmarks such as FineGym to verify whether the proposed representation can capture motion and temporal variations beyond persistent object semantics.
Questions for authors

See the weakness above.

Reviewer_fqREOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Granularity-Aligned Retrieval (GAR) for text-to-video retrieval, aiming to address the semantic granularity mismatch between text tokens and frame-level video representations. The proposed method constructs Semantic Conduits by grouping visually similar patches within frames, associating semantic prototypes across frames, and applying GMM-based soft assignment. A Transformer further contextualizes the resulting conduit tokens, which are aligned with text tokens through MeanMaxSim. Experiments on four retrieval benchmarks with two CLIP backbones demonstrate competitive performance compared with existing methods.

Critical strengths
  1. The paper identifies a meaningful granularity mismatch between textual and visual representations in text-to-video retrieval. The proposed Semantic Conduits provide a reasonable way to preserve spatially localized information while aggregating visual evidence across frames.

  2. The method is evaluated on four standard benchmarks with two CLIP backbones and achieves competitive retrieval performance. The ablation studies also demonstrate the effectiveness of the main components.

Critical weaknesses
  1. The proposed method mainly combines existing techniques, including similarity-based patch grouping, graph-based association, GMM clustering, and token-level late interaction. While the overall design is reasonable, its methodological novelty appears limited. More importantly, the paper does not provide sufficient evidence that the improvements specifically result from better semantic granularity alignment. Comparisons with simpler clustering or token aggregation methods under matched token budgets would be important to justify the proposed design.

  2. Despite the additional complexity, the improvements over Video-ColBERT are generally small. In particular, with ViT-B/16, the R@1 gains are only 0.1–0.3 percentage points across all four benchmarks. Without results from multiple random seeds or statistical analysis, it is difficult to determine whether these improvements are meaningful. Furthermore, the claimed efficiency advantages are supported mainly by analytical comparisons, without measured preprocessing costs or retrieval latency.

Questions for authors
  1. Can the authors compare Semantic Conduits with simpler alternatives, such as direct patch clustering or token merging, under the same token budget and matching framework? This would help clarify whether the gains come from the proposed semantic association mechanism or from visual token aggregation in general.

  2. In Eq. (3), conduit tokens are obtained by averaging patch features across frames, while the temporal bias in Eq. (5) only captures temporal overlap. How does the method preserve temporal ordering and motion information? Can the authors provide experiments on temporally sensitive retrieval queries?

  3. Given the marginal improvements over Video-ColBERT, can the authors report results across multiple random seeds? Additionally, can the authors provide actual preprocessing and retrieval latency measurements to support the claimed efficiency benefits?

Reviewer_zYGnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that text and video differ in information density, and that this creates a mismatch in semantic granularity between text tokens and the visual units used for matching. Global methods compress a video into a single embedding. Frame-level local-matching methods compress each frame into one feature. In both cases, several visual concepts are merged into one unit, which is then compared with text tokens that each carry much more localized meaning. To address this, the authors propose Granularity-Aligned Retrieval (GAR). It builds a hierarchical patch–region–conduit representation in five steps:

  1. Region prototypes. Within each frame, patches whose cosine affinity exceeds a threshold are grouped into connected components. Each retained component is averaged into a region prototype.

  2. Initial conduits. Prototypes receive a sinusoidal temporal encoding and are linked across adjacent frames. The resulting cross-frame connected components form the initial Semantic Conduits; their number determines the conduit count M.

  3. Soft aggregation. A per-video diagonal-covariance GMM, initialized from the conduit means, softly assigns all video patches to conduits. The responsibilities are used to pool CLIP patch features into conduit tokens.

  4. Contextualization. A Transformer with a temporal-IoU attention bias contextualizes the conduit tokens.

  5. Matching. Text tokens are matched to conduit tokens with MeanMaxSim, a ColBERT-style late interaction. Because the video representation does not depend on the query, it can be precomputed. The authors report results on MSR-VTT, MSVD, DiDeMo, and ActivityNet Captions with CLIP ViT-B/32 and ViT-B/16 backbones. GAR improves R@1 over Video-ColBERT and also reduces the number of cached visual tokens (an average of 6.6 per video on MSR-VTT with ViT-B/32).

Critical strengths
  1. Clear and well-motivated perspective. The paper explicitly frames text-to-video retrieval as a problem of matching visual units to text tokens at comparable semantic granularity. Its design principle, preserving spatial semantic diversity while consolidating evidence along the temporal dimension, is intuitive and clearly illustrated.

  2. Query-independent and compact video representation. Conduit construction and contextualization depend only on the video, so representations can be precomputed and cached. The average number of cached tokens per video (6.6 on MSR-VTT with ViT-B/32) is smaller than for frame-based late-interaction methods such as X-CLIP (13) and Video-ColBERT (26). This is practically relevant for large-scale retrieval.

  3. Broad experimental coverage and detailed implementation. Experiments span four standard benchmarks and two backbones, report nDCG@10 in addition to Recall and MnR, and include analyses of Transformer depth and of the region-selection parameters α and β. The appendix documents the implementation in considerable detail. Interpretable qualitative analysis. The case studies show that individual conduits can correspond to distinct entities and to the background, and that noun tokens such as "man," "woman," "dog," and "beach" align with them in a plausible way. The authors also openly discuss limitations, such as the imprecise localization of action words.

Critical weaknesses

W1. Insufficient positioning of the contribution and comparison with related methods

The main components of GAR all have relevant precedents: visual token aggregation, cross-frame information integration, and token-level late interaction. Although the paper discusses some prior work, it does not sufficiently explain what the hierarchical conduit construction contributes beyond existing prototype-matching and multi-granularity alignment methods. In particular, two closely related works are neither discussed nor compared against: ● ProST [1] performs hierarchical video–text matching through spatial object prototypes and temporal event prototypes. ● UCoFiA [2] jointly performs cross-modal alignment at the patch, frame, and video levels. Both are directly relevant to the core motivation and representational organization of this paper. The authors should clarify how GAR differs from them in prototype construction, cross-frame aggregation, and matching mechanism, and should provide corresponding comparative evidence. The experiments also omit other relevant methods, including HBI [3], UATVR [4], DiCoSA [5], and T-MASS [6]: ● T-MASS reports 50.2% R@1 with ViT-B/32 and 52.7% with ViT-B/16 on MSR-VTT. ● UCoFiA reports 49.4% R@1, a result that includes Sinkhorn-Knopp normalization. These methods differ in query dependence, sampling, and post-processing. I therefore suggest either including their results with the comparison conditions clearly annotated, or explaining why they fall outside the scope of this comparison. The paper should also distinguish between results quoted from the literature and results reproduced by the authors. GAR's reported preprocessing and main training configuration match those of Video-ColBERT [7], but the two differ in loss function, query expansion, and matching branches. These design differences do not by themselves make the comparison unfair. They do mean, however, that the performance gap cannot be attributed solely to the conduit representation. A reproduction under a unified pipeline, together with controlled comparisons, would allow a more accurate assessment of the contribution.

W2. Small improvements over the closest baseline, with insufficient evidence of stability

According to Tables 2 and 3, GAR improves R@1 over Video-ColBERT by only 0.1–0.6 percentage points in seven of the eight dataset/backbone settings. The paper reports no means or standard deviations over multiple random seeds and no paired bootstrap or other statistical analysis. It is therefore unclear whether these small advantages are stable. In addition, Figure 4 shows that R@1 ranges from 44.2% to 48.5% across α/β configurations, so the compression configuration has a substantial effect on performance. Appendices B.2 and B.3 justify the final hyperparameters using the reported results, but they do not say whether the test set was involved in hyperparameter selection, nor what criterion determined the final model configuration. This sensitivity does not by itself bear on statistical significance. Combined with the small margins over the baseline, however, it makes repeated runs and a clear account of parameter selection necessary to support the reliability of the comparison.

W3. Current ablations do not establish the independent contribution of the GMM and of conduit initialization

The existing ablations show that GMM refinement and soft assignment are effective together. They do not show whether the GMM is better than other clustering schemes, or whether initialization from cross-frame conduits is necessary. I recommend the following controlled comparisons, all using the same encoder, training configuration, downstream Transformer, and output token budget: ● GMM versus alternatives such as K-means; ● conduit-based mean initialization versus standard initialization; ● applying a GMM directly to all video patches, without the graph-based stages. In the initialization ablation, the number of components per video should be held fixed, so that the effect of the initialization strategy can be separated from the effect of the number of components. These comparisons would show whether the gains come from the GMM itself, from the initialization provided by the cross-frame graph structure, or from patch aggregation more generally.

W4. Efficiency evaluation lacks measured comparison of video representation construction

The paper quantifies cached embedding size and the number of similarity computations needed for query–video matching. It does not compare the actual cost of the full video representation pipeline. GAR adds intra-frame affinity computation, graph construction, connected-component extraction, and per-video GMM fitting. The authors should therefore report: ● offline construction time, throughput, and memory overhead relative to methods such as Video-ColBERT; ● how often the GMM is fitted during training, and how this affects training throughput. Moreover, fewer similarity computations do not necessarily translate into faster online retrieval. In its experimental setting, Video-ColBERT observed that single-branch and dual-branch matching had very similar query latencies even though they required different numbers of dot products [7, Appendix, Sec. 9]. The reported reduction in cache size is valuable, but measuring actual query latency under unified hardware, gallery size, and batching conditions is needed to fully assess the overall efficiency gains. [1] P. Li et al. Progressive Spatio-Temporal Prototype Matching for Text-Video Retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [2] Z. Wang et al. Unified Coarse-to-Fine Alignment for Video-Text Retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [3] P. Jin et al. Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. [4] B. Fang et al. UATVR: Uncertainty-Adaptive Text-Video Retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [5] P. Jin et al. Text-Video Retrieval with Disentangled Conceptualization and Set-to-Set Alignment. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2023. [6] J. Wang et al. Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [7] A. Reddy et al. Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19691–19701, 2025. arXiv:2503.19009.

Questions for authors
  1. GMM posteriors and fitting stability. Under the ViT-B/32, 12-frame setting, a diagonal-covariance GMM is fitted per video to 588 patch observations of dimension 512. Does this produce highly concentrated posteriors, or components with few effective samples? Please report: ○ the distribution of responsibility entropy or of maximum posterior probability; ○ the effective sample size per component; ○ convergence behavior. These would clarify how soft the assignment actually is and how stable the fitting is.

  2. Role of temporal encoding in cross-frame similarity. When linking prototypes across adjacent frames, all candidates come from the same frame. The temporal encoding therefore cannot distinguish candidates by temporal distance, although its cross terms and normalization effects may still change the ranking and threshold decisions of the cosine similarity. Please explain the rationale for adding temporal encoding at this stage, and evaluate its effect on association quality and on the link acceptance rate.

  3. Choice and transferability of rule-based hyperparameters. How were τc and τa chosen, and how was the minimum component span chosen (at least 2 frames for T = 12, at least 4 frames for T = 64)? How sensitive are the results to these values, and how should they be adjusted when video length or sampling density changes?

  4. Basis for hyperparameter and final model selection. Appendices B.2 and B.3 justify the Transformer depth and the choice of α and β using experimental results. Please state: ○ whether these parameters were selected on an independent validation set, on a validation subset held out from the training data, or on test-set results; ○ the rule used to select the final checkpoint; ○ whether the test set was involved in hyperparameter or checkpoint selection. If an independent validation procedure was used, please describe the data split, the evaluation metric, and the selection process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.