- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses local matching ambiguity and the challenge of supporting descriptions spanning multiple activities in multi-event video–text retrieval. It proposes a context-coupled scoring framework based on frozen pretrained representations. The framework jointly considers local-element support and the corresponding activity context when selecting matches, while allowing different text fragments to obtain evidence from different parts of the entire video, thereby enabling unified text-to-video (T2V) and video-to-text (V2T) retrieval. Experiments show that the proposed method achieves promising bidirectional retrieval performance on Charades-Event and ActivityNet-Event, with ablation studies and the Shuffle and Restricted experiments providing supporting evidence for its design rationale.
Critical strengths
- The research objective of the paper is clearly defined. Given fixed representations provided by the backbone, it investigates whether the way similarity scores are aggregated determines the model’s ability to distinguish different combinations of local and contextual relationships.
- The proposed method consists of two components: Element Matching & Calibration and Context Matching & Calibration, with their scores ultimately aggregated into a unified Fusion Score. In terms of experiments, the paper not only compares its bidirectional retrieval performance with existing methods but also conducts additional controlled experiments, including the Shuffle and Restricted comparisons.
Critical weaknesses
- Although the paper compares its method with many existing approaches in Table 1, the backbone used in this work appears to differ from those used in previous studies, making the comparison unfair. The motivation for choosing frozen InternVideo2 representations is also insufficiently explained.
- The specific procedures for visual parsing and linguistic parsing are not adequately described. It is unclear how the accuracy of these parsing steps is ensured, or whether parsing errors could significantly affect the effectiveness of the overall method.
- Regarding efficiency, although candidate videos can be parsed offline and the associated latency may therefore be disregarded, the authors mention that parsing text descriptions relies on an external API and takes approximately seven seconds. Such latency seems unacceptable for a real-time retrieval system, as users' online queries must undergo this time-consuming parsing process before encoding can even begin.
Questions for authors
See weeknesses
3 comments
W3. External-API text parsing takes approximately seven seconds, which seems incompatible with online real-time retrieval.
1. Low-latency text parsing for online T2V.
The reported 7.23 s is the original API-based text decomposition cost, not the cost of matching a query to each video. For the common text-to-video (T2V) setting, we additionally evaluated a lightweight action-based sentence-splitting script that extracts actions and divides descriptions into element clauses. It takes 130.21 ms per query on average and yields element splits consistent with the API parser for 98.2% of the evaluated descriptions. Video elements, contexts, memberships, and features can be prepared offline; the full description remains the textual context. This offers an online T2V path without an API call for each incoming text query, while preserving the same context-coupled scoring rule.
2. Stage-wise latency and what it measures.
The following measurements use the processed ActivityNet-Event gallery. The retrieval totals include per-query full-gallery loading, rather than assuming a permanently resident serving index.
| Measured stage | T2V: 4,877-video gallery | V2T: 17,497-text gallery |
|---|---|---|
| Uncached query preparation | ||
| API parsing (original pipeline) | 7.23 s | 8.71 s |
| Action-splitting script (T2V alternative) | 130.21 ms | — |
| Query feature encoding | 0.030 s | 1.270 s |
| Retrieval with precomputed structures | ||
| Gallery metadata loading/conversion | 916.74 ms | 1,932.48 ms |
| Gallery feature loading/preparation | 506.19 ms | 420.62 ms |
| Scoring, ranking, and return | 11.82 ms | 2.74 ms |
| Total retrieval including gallery loading | 1.435 s | 2.356 s |
The previously reported 34.76 s describes scoring/ranking over the entire evaluation split, not one query. With the script, the sum of measured T2V stages is approximately 1.60 s, including 1.423 s of repeated gallery loading; the original API-based stage sum is 8.70 s. If the gallery is kept in memory, text splitting, encoding, and scoring together account for approximately 172 ms of measured component time, excluding serving overhead. These are stage sums, not separately measured end-to-end deployment latencies. Crucially, the context-coupled scoring and ranking itself takes only 11.82 ms over the gallery.
3. Deployment scope and API cost.
The V2T direction starts from a video, so an unseen video still requires heavier visual parsing and encoding (12.34 s as a cold-query stage sum); it is better suited to indexed or reusable video queries. For the original API workflow, an uncached text-query parsing call costs approximately one-quarter of a video-parsing call, and batching texts can reduce repeated prompt-token costs. The lightweight T2V splitter removes that text-query API dependency altogether. ReContext's core contribution is structured, training-free ranking without per-candidate LLM inference, not a claim that every cold-query configuration is already real-time.
We welcome further questions and discussion, and hope these clarifications and additional evidence support reconsideration of the rating.
Additional Response to Reviewer Eirb
Thank you for looking closely at our backbone comparisons and for asking how the parsing works in practice. Your questions helped us clarify where the gains come from and what an online text query costs.
- Backbone choice and fairness. Frozen InternVideo2 provides a shared video–text space for training-free scoring. ReContext gains 4.54 pp in T2V R@1 over Plain Fusion with identical inputs. Gains also transfer to frozen CLIP (our reply to Reviewer wCT7, Q1, “Matched-input evidence” and “Cross-backbone evidence”). In the new Qwen3-VL-Embedding-2B experiment, ReContext adds 5.28/5.18 pp in T2V/V2T R@1 over the frozen backbone alone (wCT7 Q3, baseline table).
- Parsing and error sensitivity. We explained how we group timestamped visual elements and split text into action fragments. A 100-pair human audit found 91.0–97.9% validity across four criteria. Video-wide matching can still use valid elements after grouping errors. With Gated Shuffle as the control, correct memberships improve T2V/V2T R@1 by 0.532/0.253 pp (wCT7 Q2, full ActivityNet-Event comparison).
- Online query latency. Our 130 ms T2V splitter replaces the 7.23 s API step and matches its splits on 98.2% of descriptions. With gallery loading, our fast pipeline takes 1.63 s/query on one RTX 3090 for 4,877 videos. BLiM-7B and X-CoT report comparable times of 1.75 and 1.80 s/query on multiple GPUs (wCT7 Q1, “Is ReContext's query latency practical?”).
We would be grateful if you could read our full reply and let us know whether these additional results address your concerns. We hope you will reconsider your assessment in light of this evidence.
We thank the reviewer for the careful assessment of our framework. We clarify backbone fairness, the contributions of structured scoring, parsing reliability, and practical online retrieval latency.
1. Backbone choice and comparison fairness.
ReContext is a training-free retrieval scoring framework. We use frozen InternVideo2-CLIP-1B-224p-f8 to represent elements and contexts in a shared video–text space without task-specific adaptation. Cross-backbone, task-supervised results in Table 1 provide benchmark context, not encoder-controlled evidence. Our controlled evidence comes from scoring ablations that reuse the same parsed structures and frozen representations.
2. Improvements beyond semantic parsing.
Plain Fusion averages local and contextual evidence using the same parsing and encoder. The Fusion row in Table 4 combines separately calibrated branches without member-supported context enhancement; ReContext additionally couples each context with evidence from its own members before context aggregation.
Relative to Plain Fusion, ReContext gains +4.54 T2V and +2.27 V2T percentage points (95% CIs: [3.87, 5.21] and [1.91, 2.64]). Because parsing and representations are fixed, calibration and score aggregation explain these gains. The incremental benefit over calibrated Independent Fusion is +0.18/+0.17 points, which we distinguish from the larger overall scoring gain. Eq. (11) and the constructed configurations in Appendix C (Table 6) establish the underlying distinction: independently pooled maxima can conceal different context–member associations, whereas coupled scoring retains them.
3. Backbone transfer and ablation scope.
With frozen CLIP ViT-L/14, T2V R@1 increases from 1.56% to 2.20% and V2T R@1-Average from 1.04% to 1.36%. Absolute performance depends on the encoder, but these controlled gains show that the scoring design is not specific to InternVideo2. We conduct detailed scoring ablations on ActivityNet, whose 8,502 multi-fragment/multi-context queries support mechanism-level analysis, and retain Charades as a separate end-to-end retrieval benchmark.
1. Visual parsing. Qwen3.5-Plus extracts temporally localized visual elements and groups semantically related elements into activity contexts. Each context retains member indices, temporal locations, and a grouping rationale. Memberships may overlap or connect nonadjacent events; they need not form a disjoint temporal partition. Ground-truth temporal boundaries are never used as retrieval inputs.
2. Linguistic parsing. Descriptions undergo (i) removal of camera-related or subjective narration, (ii) atomic-action extraction, (iii) identification of temporal/causal relations, and (iv) reconstruction of interpretable fragments. Participants, actions, objects, and relevant modifiers are preserved.
3. Human evaluation of parsing quality. Two independent annotators assessed 100 ActivityNet video–text pairs with binary Valid/Invalid labels. Pooled agreement across the four dimensions was Cohen's kappa of 0.92; a third annotator adjudicated disagreements. The table reports final adjudicated validity.
4. Sensitivity to parsing errors. Omitted actions, imprecise boundaries, and incorrect memberships can reduce retrieval quality. However, the video-wide branch matches each description fragment against all visual elements, independently of the selected context. The context branch uses only its own members for one-sided enhancement, so weak member evidence does not subtract from context compatibility. A context-assignment error therefore need not suppress valid video-wide matches, although omitted semantic elements cannot be recovered. Parsing quality affects coupling without turning any single context assignment into a hard retrieval constraint.