acceptodds
Under review as a conference paper at ICLR 2027

ReContext: Context-Coupled Matching for Multi-Event Video-Text Retrieval

Abstract

In multi-event video–text retrieval, a description refers to selected parts of one or several activities within a video. Local actions and objects help locate relevant content and distinguish similar activities, while the surrounding activity provides context for judging their relevance to the description. Pooling the strongest local and contextual matches independently discards their pairing, making different local–activity associations indistinguishable when their pooled maxima coincide. For descriptions that span several activities, relevance also depends on combining support from different contexts. To this end, we introduce ReContext, which supplements each context's compatibility with the description using matches from its own constituent spans, then selects the highest-scoring context. We combine this context score with an aggregation of each description fragment's best match across the video, yielding a shared bidirectional score using frozen representations. Our analysis shows that coupled selection can retain distinctions between local–context configurations that independent pooling assigns identical local and contextual maxima. Experiments on Charades-Event and ActivityNet-Event show improved retrieval in both directions, with controlled comparisons examining the contributions of membership coupling and cross-activity support.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video-text retrieval, multi-event video analysis

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_EirbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses local matching ambiguity and the challenge of supporting descriptions spanning multiple activities in multi-event video–text retrieval. It proposes a context-coupled scoring framework based on frozen pretrained representations. The framework jointly considers local-element support and the corresponding activity context when selecting matches, while allowing different text fragments to obtain evidence from different parts of the entire video, thereby enabling unified text-to-video (T2V) and video-to-text (V2T) retrieval. Experiments show that the proposed method achieves promising bidirectional retrieval performance on Charades-Event and ActivityNet-Event, with ablation studies and the Shuffle and Restricted experiments providing supporting evidence for its design rationale.

Critical strengths
  1. The research objective of the paper is clearly defined. Given fixed representations provided by the backbone, it investigates whether the way similarity scores are aggregated determines the model’s ability to distinguish different combinations of local and contextual relationships.
  2. The proposed method consists of two components: Element Matching & Calibration and Context Matching & Calibration, with their scores ultimately aggregated into a unified Fusion Score. In terms of experiments, the paper not only compares its bidirectional retrieval performance with existing methods but also conducts additional controlled experiments, including the Shuffle and Restricted comparisons.
Critical weaknesses
  1. Although the paper compares its method with many existing approaches in Table 1, the backbone used in this work appears to differ from those used in previous studies, making the comparison unfair. The motivation for choosing frozen InternVideo2 representations is also insufficiently explained.
  2. The specific procedures for visual parsing and linguistic parsing are not adequately described. It is unclear how the accuracy of these parsing steps is ensured, or whether parsing errors could significantly affect the effectiveness of the overall method.
  3. Regarding efficiency, although candidate videos can be parsed offline and the associated latency may therefore be disregarded, the authors mention that parsing text descriptions relies on an external API and takes approximately seven seconds. Such latency seems unacceptable for a real-time retrieval system, as users' online queries must undergo this time-consuming parsing process before encoding can even begin.
Questions for authors

See weeknesses

3 comments

We thank the reviewer for the careful assessment of our framework. We clarify backbone fairness, the contributions of structured scoring, parsing reliability, and practical online retrieval latency.

W1. Table 1 compares different backbones, potentially compromising fairness; why choose frozen InternVideo2?

1. Backbone choice and comparison fairness.

ReContext is a training-free retrieval scoring framework. We use frozen InternVideo2-CLIP-1B-224p-f8 to represent elements and contexts in a shared video–text space without task-specific adaptation. Cross-backbone, task-supervised results in Table 1 provide benchmark context, not encoder-controlled evidence. Our controlled evidence comes from scoring ablations that reuse the same parsed structures and frozen representations.

2. Improvements beyond semantic parsing.

Plain Fusion averages local and contextual evidence using the same parsing and encoder. The Fusion row in Table 4 combines separately calibrated branches without member-supported context enhancement; ReContext additionally couples each context with evidence from its own members before context aggregation.

ConfigurationT2V R@1 (%)V2T R@1-Average (%)
InternVideo2 baseline17.418.30
Plain Fusion19.668.81
Independent Fusion (Table 4)24.0210.91
ReContext24.2011.08

Relative to Plain Fusion, ReContext gains +4.54 T2V and +2.27 V2T percentage points (95% CIs: [3.87, 5.21] and [1.91, 2.64]). Because parsing and representations are fixed, calibration and score aggregation explain these gains. The incremental benefit over calibrated Independent Fusion is +0.18/+0.17 points, which we distinguish from the larger overall scoring gain. Eq. (11) and the constructed configurations in Appendix C (Table 6) establish the underlying distinction: independently pooled maxima can conceal different context–member associations, whereas coupled scoring retains them.

3. Backbone transfer and ablation scope.

With frozen CLIP ViT-L/14, T2V R@1 increases from 1.56% to 2.20% and V2T R@1-Average from 1.04% to 1.36%. Absolute performance depends on the encoder, but these controlled gains show that the scoring design is not specific to InternVideo2. We conduct detailed scoring ablations on ActivityNet, whose 8,502 multi-fragment/multi-context queries support mechanism-level analysis, and retain Charades as a separate end-to-end retrieval benchmark.


W2. How are visual and linguistic structures parsed, how is their accuracy evaluated, and can parsing errors undermine retrieval?

1. Visual parsing. Qwen3.5-Plus extracts temporally localized visual elements and groups semantically related elements into activity contexts. Each context retains member indices, temporal locations, and a grouping rationale. Memberships may overlap or connect nonadjacent events; they need not form a disjoint temporal partition. Ground-truth temporal boundaries are never used as retrieval inputs.

2. Linguistic parsing. Descriptions undergo (i) removal of camera-related or subjective narration, (ii) atomic-action extraction, (iii) identification of temporal/causal relations, and (iv) reconstruction of interpretable fragments. Participants, actions, objects, and relevant modifiers are preserved.

3. Human evaluation of parsing quality. Two independent annotators assessed 100 ActivityNet video–text pairs with binary Valid/Invalid labels. Pooled agreement across the four dimensions was Cohen's kappa of 0.92; a third annotator adjudicated disagreements. The table reports final adjudicated validity.

DimensionValid / TotalValidity
Textual semantic fidelity95 / 10095.0%
Visual element grounding461 / 47197.9%
Context semantic coherence253 / 27891.0%
Context–element membership validity442 / 47493.2%

4. Sensitivity to parsing errors. Omitted actions, imprecise boundaries, and incorrect memberships can reduce retrieval quality. However, the video-wide branch matches each description fragment against all visual elements, independently of the selected context. The context branch uses only its own members for one-sided enhancement, so weak member evidence does not subtract from context compatibility. A context-assignment error therefore need not suppress valid video-wide matches, although omitted semantic elements cannot be recovered. Parsing quality affects coupling without turning any single context assignment into a hard retrieval constraint.

W3. External-API text parsing takes approximately seven seconds, which seems incompatible with online real-time retrieval.

1. Low-latency text parsing for online T2V.

The reported 7.23 s is the original API-based text decomposition cost, not the cost of matching a query to each video. For the common text-to-video (T2V) setting, we additionally evaluated a lightweight action-based sentence-splitting script that extracts actions and divides descriptions into element clauses. It takes 130.21 ms per query on average and yields element splits consistent with the API parser for 98.2% of the evaluated descriptions. Video elements, contexts, memberships, and features can be prepared offline; the full description remains the textual context. This offers an online T2V path without an API call for each incoming text query, while preserving the same context-coupled scoring rule.

2. Stage-wise latency and what it measures.

The following measurements use the processed ActivityNet-Event gallery. The retrieval totals include per-query full-gallery loading, rather than assuming a permanently resident serving index.

Measured stageT2V: 4,877-video galleryV2T: 17,497-text gallery
Uncached query preparation
API parsing (original pipeline)7.23 s8.71 s
Action-splitting script (T2V alternative)130.21 ms—
Query feature encoding0.030 s1.270 s
Retrieval with precomputed structures
Gallery metadata loading/conversion916.74 ms1,932.48 ms
Gallery feature loading/preparation506.19 ms420.62 ms
Scoring, ranking, and return11.82 ms2.74 ms
Total retrieval including gallery loading1.435 s2.356 s

The previously reported 34.76 s describes scoring/ranking over the entire evaluation split, not one query. With the script, the sum of measured T2V stages is approximately 1.60 s, including 1.423 s of repeated gallery loading; the original API-based stage sum is 8.70 s. If the gallery is kept in memory, text splitting, encoding, and scoring together account for approximately 172 ms of measured component time, excluding serving overhead. These are stage sums, not separately measured end-to-end deployment latencies. Crucially, the context-coupled scoring and ranking itself takes only 11.82 ms over the gallery.

3. Deployment scope and API cost.

The V2T direction starts from a video, so an unseen video still requires heavier visual parsing and encoding (12.34 s as a cold-query stage sum); it is better suited to indexed or reusable video queries. For the original API workflow, an uncached text-query parsing call costs approximately one-quarter of a video-parsing call, and batching texts can reduce repeated prompt-token costs. The lightweight T2V splitter removes that text-query API dependency altogether. ReContext's core contribution is structured, training-free ranking without per-candidate LLM inference, not a claim that every cold-query configuration is already real-time.


We welcome further questions and discussion, and hope these clarifications and additional evidence support reconsideration of the rating.

AuthorsOpenReview ↗

Additional Response to Reviewer Eirb

Thank you for looking closely at our backbone comparisons and for asking how the parsing works in practice. Your questions helped us clarify where the gains come from and what an online text query costs.

  • Backbone choice and fairness. Frozen InternVideo2 provides a shared video–text space for training-free scoring. ReContext gains 4.54 pp in T2V R@1 over Plain Fusion with identical inputs. Gains also transfer to frozen CLIP (our reply to Reviewer wCT7, Q1, “Matched-input evidence” and “Cross-backbone evidence”). In the new Qwen3-VL-Embedding-2B experiment, ReContext adds 5.28/5.18 pp in T2V/V2T R@1 over the frozen backbone alone (wCT7 Q3, baseline table).
  • Parsing and error sensitivity. We explained how we group timestamped visual elements and split text into action fragments. A 100-pair human audit found 91.0–97.9% validity across four criteria. Video-wide matching can still use valid elements after grouping errors. With Gated Shuffle as the control, correct memberships improve T2V/V2T R@1 by 0.532/0.253 pp (wCT7 Q2, full ActivityNet-Event comparison).
  • Online query latency. Our 130 ms T2V splitter replaces the 7.23 s API step and matches its splits on 98.2% of descriptions. With gallery loading, our fast pipeline takes 1.63 s/query on one RTX 3090 for 4,877 videos. BLiM-7B and X-CoT report comparable times of 1.75 and 1.80 s/query on multiple GPUs (wCT7 Q1, “Is ReContext's query latency practical?”).

We would be grateful if you could read our full reply and let us know whether these additional results address your concerns. We hope you will reconsider your assessment in light of this evidence.

Reviewer_hPrpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multi-event video–text retrieval. A local action or object may look relevant but belong to the wrong activity. The authors argue that picking the strongest local and context matches separately can combine evidence from different activities and lose the local–context association. ReContext uses Qwen3.5-Plus for semantic parsing and frozen InternVideo2 features. It combines local evidence with its own context through membership coupling, while keeping video-wide support so that different activities can support different parts of a description. The method improves retrieval in both directions on two datasets.

Critical strengths
  1. Similar local actions can mean different things in different activities. Framing this through local–context association makes sense, and keeping the encoder fixed helps compare the scoring choices.
  2. Keeping video-wide support fits the multi-event setting. Results in both retrieval directions on two datasets, along with targeted controls, support this design.
Critical weaknesses
  1. On ActivityNet-Event, T2V R@1 rises from 24.02% for Fusion to 24.20% for ReContext, only 0.18 percentage points. The Shuffle experiment also shows no overall advantage from keeping the original pairing, although some subsets benefit. I find the evidence for parsing and context more convincing than the evidence that membership is a key source of the overall improvement. The paper already includes some confidence intervals, but the stability of the ReContext–Fusion gain still needs to be shown directly.

  2. The main text uses max operations, while the appendix implementation uses top-three means. A complete scoring rule matching the results is needed. API parsing also takes about seven seconds per description, and calibration depends on the full evaluation gallery. The paper reports some timing measurements, but the latency and cost of a new query, and ranking stability when the gallery changes, remain unclear. Given the small coupling gain, whether these extra costs are worthwhile matters.

Questions for authors
  1. Can you give one consistent description of the scoring rule used in the experiments? With parsing, features, and aggregation held fixed, how stable is the gain from membership coupling across both datasets and retrieval directions? Please include confidence intervals for this comparison.
  2. How much do retrieval results change when the reference pool changes, or when videos are added to or removed from the gallery?
  3. For an uncached text query, what are the total time and API cost from parsing to the final ranking? Compared with Fusion using the same parsed inputs, how much accuracy and cost does ReContext add?
4 comments

We thank the reviewer and will clarify coupling with full-benchmark gated and shuffled controls, report confidence intervals and query costs, and specify the exact Top-3 implementation.

W1. Coupling adds only +0.18 pp T2V R@1; Shuffle shows no overall pairing advantage.

Q1 (stability). Is coupling stable across datasets and retrieval directions with other components fixed? What are the confidence intervals?

1. Reliability-gated coupling: when genuine membership matters.

The original unconditional coupling yields a modest T2V gain over Fusion (+0.18 pp) and no overall advantage over Shuffle (−0.04 pp). Our diagnostics identify the reason: member support helps when it agrees with its containing context, but may be misleading under weak alignment. We test this directly with a reliability-gated ReContext variant, activating the original coupling only for pairs with aligned evidence and reliable membership structure.

Let CkC_k be calibrated context compatibility, EkE_k its own members' support, and IkI_k its member indices. Define

Q=\frac{\sum_k B_kq_k}{\sum_k B_k+\epsilon},$$ $$q_k=0.30\min(1,|I_k|/3)+0.20q_{\mathrm{temp},k}+0.50q_{\mathrm{vis},k},$$ $$g(V,T)=\mathbf1!\left[\max_k B_k>0;\land;0.65A+0.35Q\ge0.60\right],\quad S_{\mathrm{gate}}=F+g(U_\beta-F).$$ Here $\widetilde C_k,\widetilde E_k$ use fixed training-split percentile normalization; $q_{\mathrm{temp}}$ and $q_{\mathrm{vis}}\in[0,1]$ measure temporal coherence and visual member-feature consistency. **The max in $A$ tests whether at least one activity context is jointly supported by its context and members** (the min requires both); averaging across unrelated activities would obscure valid partial-video relevance. $Q$ weights structural reliability only where members can enhance their context. $F$ is Fusion and $U_\beta$ is original ReContext. All gate weights/thresholds were selected on the training split and frozen for testing; encoders, original calibration/aggregation, and coupling parameters remain fixed. No extra API calls, encoder passes, or gradient training are needed. **Full ActivityNet-Event benchmark (R@1, %):** | Method | T2V | V2T-Avg. | | ----------------------------------------- | --------------- | --------------- | | Baseline | 17.41 | 8.30 | | _**Plain Fusion**_ | 19.66 | 8.81 | | Fusion | 24.02 | 10.91 | | ReContext | 24.20 | 11.08 | | **_Gated + Shuffle_** | 24.17 | 11.03 | | _**Reliability-gated ReContext**_ | **24.70** | **11.29** | Against Fusion, gating improves **T2V by +0.68 pp [95% CI: +0.46, +0.90]** and **V2T by +0.37 pp [+0.25, +0.51]**. Crucially, against **gated Shuffle**, the genuine pairings improve T2V by **+0.5320 pp [+0.3816, +0.6851]** and V2T by **+0.2531 pp [+0.1482, +0.3604]** (unrounded differences). **The gate is recomputed for every shuffled membership assignment** using unchanged training-derived thresholds; original-membership gate decisions are never reused. Thus, the scoring procedure is identical except for the membership pairing. Both intervals exclude zero on the **complete evaluation set**, without selecting favorable queries. Other original controls explain why gating matters: the 8,502 multi-fragment/context queries show −0.09 pp [−0.29, 0.10] versus Shuffle. ReContext − Shuffle is **+1.05 pp [0.53, 1.58]** on 1,758 aligned queries and **+2.38 pp [1.25, 3.66]** on the 442 stronger-enhancement queries, but **−1.78 pp [−3.28, −0.51]** on 264 low-alignment queries. The resulting conclusion is conditional,: reliable pairing provides useful evidence, while unconditional enhancement can also propagate misaligned support. **2. Framework contribution versus the incremental coupling term.** The +0.18 pp comparison isolates only _original coupling_, because Fusion and ReContext otherwise share parsed inputs, representations, calibration, and fusion weights. Relative to **Plain Fusion on the same parsed inputs**, original ReContext gains **+4.54 pp T2V [3.87, 5.21]** and **+2.28 pp V2T [1.91, 2.64]** (95% CIs), demonstrating the broader value of structured score composition. The isolated original ReContext − Fusion differences are **+0.1844 pp T2V [−0.0912, 0.4623]** and **+0.1704 pp V2T [0.0144, 0.3231]**; only the latter interval excludes zero. On Charades-Event, original ReContext also exceeds frozen InternVideo2 in both directions (T2V **2.87→3.17**; V2T-Avg. **2.04→2.19**). Those Charades results establish full-framework improvement. The gated full-benchmark control now directly establishes the contribution of **genuine memberships** on ActivityNet-Event in both directions.

W2 / Q1 (scoring). Main-text maxima differ from Appendix B's Top-3 means; which rule generated the results?

The original max-only formulation was intended to emphasize coupling before pooling, whereas the reported implementation uses Top-3 aggregation to reduce dependence on individual extreme responses. We recognize that presenting the simplified formulation without explicitly distinguishing it from the implemented operator was insufficiently clear.

We have revised Fig. 2 (p. 4), Eqs. (7)–(10) (p. 5), and the corresponding analysis in Appendix C (p. 17) to explicitly distinguish the implemented aggregation from the simplified formulation.

For calibrated element scores Aij=sˉijeA_{ij}=\bar s^e_{ij}, context scores Ck=sˉkcC_k=\bar s^c_k, and membership sets IkI_k: Lk=max⁡jTop3Meani∈Ik(Aij),L_k=\max_j Top3Mean_{i\in I_k}(A_{ij}),

Hk=Ck+0.6[Lk−Ck]+,H_k=C_k+0.6[L_k-C_k]_+,

Sc=0.25max⁡kHk+0.75Top3Meank(Hk),S_c=0.25\max_k H_k+0.75 Top3Mean_k(H_k),

Uj=max⁡iAij,Se=0.85max⁡jUj+0.15Meanj(Uj),U_j=\max_i A_{ij},\quad S_e=0.85\max_jU_j+0.15 Mean_j(U_j),

S=0.2Se+0.8Sc.S=0.2S_e+0.8S_c. Top-3 uses all available entries if fewer than three exist; videos without a valid context use SeS_e. These are the reported operators, also listed in Appendix B (p. 14).

We additionally compare the implemented Top-3 aggregation against Max-only aggregation on ActivityNet-Event.

KT2V Top-3T2V max-onlyV2T-Avg. Top-3V2T-Avg. max-only
124.2024.1211.0810.98
547.0747.2128.0527.83
1059.0358.8837.4638.02
5081.3881.4865.4166.48

Top-3 slightly improves R@1 in both directions but has mixed higher-K effects. The key contribution is coupling before pooling.


Q2 (reference pool and gallery updates). How does the reference pool affect performance? What happens when gallery videos are added or removed?

1. Calibration.

Eq. (6) normalizes each text component against a fixed visual reference. Element/context pools have different scales, so fusion is parameterized. τe,τc\tau_e,\tau_c govern concentration; β,λ\beta,\lambda govern enhancement/fusion. All were selected on 100 training-split videos and frozen for both benchmarks and these tests.

2. Reference-pool test: fixed gallery/parameters; three seeds, mean ± SD:

Reference poolT2V R@1 (%)V2T R@1-Avg. (%)
100%24.201711.0829
75%24.1825 ± 0.012011.0749 ± 0.0097
50%24.1748 ± 0.008811.0684 ± 0.0079

Halving the pool changes recall by only −0.027 pp T2V / −0.015 pp V2T. For gallery changes, fixed R\mathcal R preserves existing candidate scores and relative order; added/removed candidates may still alter ranks and recall. This test varies the reference pool.


W2 / Q3 (efficiency). What are uncached query latency/API costs through final ranking, and the incremental accuracy/cost of coupling versus Fusion?

1. Latency.

The API builds query structures, gallery features are cached. Full ActivityNet-Event timings:

StageT2V (4,877 videos)V2T (17,497 texts)
Query API parsing (original)7.23 s8.71 s
Query encoding0.030 s1.270 s
Cached metadata load/conversion916.74 ms1,932.48 ms
Feature load/preparation506.19 ms420.62 ms
Scoring, ranking, return11.82 ms2.74 ms
Total cached retrieval1.435 s2.356 s
Uncached with original API parsing≈8.70 s≈12.34 s

An action-aware clause splitter matches the API's element segmentation on 98.2% of tested texts at 130.21 ms/query. Replacing the parsing stage gives a component-sum estimate of ≈1.60 s T2V versus ≈8.70 s with API parsing. This path removes text-query parsing API calls; V2T retains video-query API parsing.

2. Cost and incremental coupling.

Under experimental pricing, API parsing costs ≈$0.00045/text or ≈$0.0018/video; ActivityNet video-gallery parsing costs ≈$9 once (GPU excluded). Cached queries incur no parsing charge. Fusion, ReContext, and gating reuse parsing, features, and calibration: coupling/gating adds no API calls or encoder passes, only score operations. Versus Fusion, original coupling gains +0.18/+0.17 pp (T2V/V2T R@1), and gating +0.68/+0.37 pp. The 11.82/2.74 ms figures measure total scoring.


We welcome further questions and discussion.

And we sincerely hope this clarification and the additional controlled evidence support reconsideration of the score.

I am raising my rating from 1 to 2. The scoring clarification and gated Shuffle results address several concerns and provide stronger evidence for membership coupling. However, the original coupling gain remains small, and the stronger results rely on a new gated variant tested only on ActivityNet-Event. I therefore remain at weak rejection.

AuthorsOpenReview ↗

Further Response to Reviewer hPrp (Coupling Gain Magnitude and Cross-Dataset Validation)

Thank you for taking the time to discuss the paper with us and for raising your rating after considering our response. We appreciate how clearly you explained the concerns that remain.

  • Original coupling gain. The gain is comparable to published gains for individual components over their respective baselines and contributes to the larger improvement of the full ReContext pipeline.
  • Gated coupling on a second dataset. Our new Charades-Event evaluation shows that gating improves both T2V and V2T retrieval.

Q1: Magnitude and Value of Context–Member Coupling

We compare component-level R@1 improvements reported in established video retrieval studies using ActivityNet benchmarks:

MethodComponentActivityNet T2V R@1 Gain
MeVTR (ICCV'23)Key Event Selection+0.14 pp
HLFormer (ICCV'25)Partial Order Preservation Loss+0.10 pp
HLFormer (ICCV'25)Query Diversity Loss+0.20 pp
DreamPRVR (CVPR'26)LpvsL_{\mathrm{pvs}}+0.10 pp
DreamPRVR (CVPR'26)Global Registers+0.20 pp
ReContext (Original)Context–Member Coupling+0.20 pp

These published ablations provide component-level reference points across related retrieval settings. The original coupling gain is consistent with established component-level improvements, while reliability gating further strengthens its contribution.

More importantly, coupling introduces complementary relational information beyond independent Fusion. Fusion already incorporates both context and local evidence, whereas ReContext explicitly preserves their membership relationships. On ActivityNet-Event, preserving genuine context–member associations improves over Gated Shuffle by +0.532 pp T2V (95% CI [0.382, 0.685]) and +0.253 pp V2T ([0.148, 0.360]). These controlled results further validate the discriminative value of the context–member associations modeled by coupling, beyond simply combining multi-granular features.

Q2: Generalization Across Datasets and Retrieval Directions

We further evaluated reliability-gated coupling on Charades-Event, demonstrating consistent improvements in both T2V and V2T retrieval. The following table places these gains in the context of published Charades-Event results:

Method / ComparisonT2V R@1 (%)T2V Δ (pp)T2V Rel. ΔV2T R@1-Avg (%)V2T Δ (pp)V2T Rel. Δ
MeVTR avg → max1.88 → 2.02+0.14+7.45%1.39 → 1.01−0.38−27.34%
CenterCLIP → MeVTR max1.93 → 2.02+0.09+4.66%1.32 → 1.01−0.31−23.48%
CenterCLIP → MeVTR avg1.93 → 1.88−0.05−2.59%1.32 → 1.39+0.07+5.30%
ReContext Full → Gated3.17 → 3.33+0.16+5.08%2.19 → 2.31+0.12+5.43%

Charades-Event is a challenging multi-event retrieval benchmark, where established methods achieve T2V R@1 scores of approximately 1–3%. Accordingly, relative improvements provide important context for interpreting absolute gains.

Our gated coupling achieves +5.08% T2V (+0.1613 pp) and +5.43% V2T (+0.1190 pp), with positive 95% confidence intervals of [0.0271, 0.3272] pp and [0.0190, 0.2529] pp, respectively. Notably, while MeVTR's avg-to-max aggregation improves T2V with a corresponding V2T trade-off, ReContext achieves consistent improvements in both retrieval directions. These results establish the effectiveness of reliability-gated coupling beyond ActivityNet-Event and across different retrieval settings.


The Charades-Event comparison complements the ActivityNet-Event membership control by showing that the gated variant also improves retrieval on a second benchmark.

We hope these results and the component-level comparisons address your remaining concerns and would be grateful if you could reconsider your assessment in light of the new evidence

If anything still stands in the way of a higher score, we would be grateful if you could point us to it so we can address it before the discussion period closes.

Reviewer_MsqJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

ReContext combines local video–text matches with activity context using frozen representations. It links each context to support from its own members while allowing description fragments to match across the video. The paper evaluates bidirectional retrieval and studies the effects of membership and support scope.

Critical strengths
  1. Separating context selection from video-wide support is sensible for descriptions spanning multiple activities. The comparison with restricted support shows a 0.51-point overall T2V R@1 gain, with a positive confidence interval, supporting this part of the design.
  2. The controlled comparisons reuse the same representations, and the paper reports negative as well as positive subgroup results. The full system improves over the flat frozen-encoder baseline in both directions, particularly on ActivityNet-Event, without task-specific encoder training.
Critical weaknesses
  1. The analyzed operator differs from the evaluated one. Equations (7) and (9) use member and context maxima, while Appendix B uses top-three member averages and a maximum/top-three mixture across contexts. These changes alter how membership and context count affect the score. The analysis of max-only coupling therefore does not directly characterize the reported method, and the corresponding max-only result is absent.
  2. Equation (8) compares weights normalized over different reference pools as though their magnitudes were commensurate. Element and context denominators depend on different pool sizes, temperatures, and score distributions. Changing the reference contexts can therefore switch enhancement on or off without changing a candidate's underlying matches. Keeping the gallery fixed makes evaluation consistent, but does not resolve this dependence of coupling on score scale.
  3. The enhancement provides positive support rather than a consistency check. Weak member evidence never lowers a context score, and the strongest description fragment controls the boost. A salient but incidental detail can therefore promote a weakly compatible activity; rejecting it relies on the frozen whole-description representation. This limits the mechanism's claimed role, particularly given the small gain over independent fusion (24.02% to 24.20% T2V R@1).
Questions for authors
  1. How does the max-only formulation perform, and which conclusions extend to the implemented top-three aggregators?
  2. What justifies comparing element and context weights in Eq. (8), and how stable is selection when the reference pool or its parsing granularity changes?
  3. How does the one-sided enhancement handle strong local matches from an activity that is incompatible with the complete description?
4 comments

We thank the reviewer for examining the scoring operators and calibration assumptions. We clarify the exact Top-3 implementation, its theoretical scope, and the robustness and efficiency of preprocessing.

W1 / Q1. The analyzed max-only operator differs from the evaluated Top-3 aggregators. How does max-only perform, and which conclusions extend to Top-3?

1. Exact implementation and correction.

The original max-only formulation was intended to emphasize coupling before pooling, whereas the reported implementation uses Top-k aggregation to reduce dependence on individual extreme responses. We recognize that presenting the simplified formulation without explicitly distinguishing it from the implemented operator was insufficiently clear.

We have revised Fig. 2 (p. 4), Eqs. (7)–(10) (p. 5), and the corresponding analysis in Appendix C (p. 17) to explicitly distinguish the implemented aggregation from the simplified formulation.

For calibrated element scores Aij=sˉijeA_{ij}=\bar s^e_{ij}, context scores Ck=sˉkcC_k=\bar s^c_k, and membership sets IkI_k:

Lk=max⁡jTop3Meani∈Ik(Aij),L_k=\max_j Top3Mean_{i\in I_k}(A_{ij}),

Hk=Ck+0.6[Lk−Ck]+,H_k=C_k+0.6[L_k-C_k]_+,

Sc=0.25max⁡kHk+0.75Top3Meank(Hk),S_c=0.25\max_k H_k+0.75 Top3Mean_k(H_k),

Uj=max⁡iAij,Se=0.85max⁡jUj+0.15Meanj(Uj),U_j=\max_i A_{ij},\quad S_e=0.85\max_jU_j+0.15 Mean_j(U_j),

S=0.2Se+0.8Sc.S=0.2S_e+0.8S_c.

Top-3 uses all available entries if fewer than three exist; videos without a valid context use SeS_e. These are the reported operators, also listed in Appendix B (p. 14).

We additionally compare the implemented Top-3 aggregation against Max-only aggregation on ActivityNet-Event.

R@KT2V Top-3T2V Max-onlyV2T Top-3V2T Max-only
124.2024.1211.0810.98
547.0747.2128.0527.83
1059.0358.8837.4638.02
5081.3881.4865.4166.48

Top-3 slightly improves R@1 in both directions but has mixed higher-K effects. The key contribution is coupling before pooling, not Top-3 itself.

W2 / Q2. Element and context weights use different reference pools. Why compare them in Eq. (8) of the original submission, and how stable are results under reference-pool or parsing-granularity changes?

1. Calibration and parameter selection.

Reference-based softmax calibration measures candidate-relative matching strength rather than assuming intrinsically identical element and context scales. Generic description fragments can produce high similarities across many videos without providing discriminative evidence. Eq. (6) normalizes each textual component against its corresponding visual reference pool, emphasizing distinctive candidate responses.

We recognize that the two pools differ in size and score distribution. Their interaction in Eq. (9) is therefore a parameterized scoring design rather than an assumption of inherently comparable semantic probabilities. Separate temperatures τe\tau_e and τc\tau_c control the calibrated response distributions, while β\beta controls the enhancement magnitude. We selected these parameters on 100 videos sampled from the training split and applied them unchanged to the full ActivityNet-Event and Charades-Event evaluations, without evaluation-set tuning. For a new retrieval collection, the same parameter-selection procedure can be performed on a representative training or development subset.

2. Reference-pool robustness.

We reduced only the calibration reference pool, keeping the candidate gallery, features, and scoring parameters fixed (three seeds; mean ±\pm SD):

Reference poolT2V R@1 (%)V2T R@1-Avg. (%)
100%24.201711.0829
75%24.1825 ±\pm 0.012011.0749 ±\pm 0.0097
50%24.1748 ±\pm 0.008811.0684 ±\pm 0.0079

At 50%, the drops are only 0.027 pp T2V and 0.015 pp V2T, without retuning. This demonstrates retrieval-level stability for the tested reference changes, not invariance of context selection or arbitrary visual reparsing.

We further clarify its partial-relevance role and present reliability-gating and membership-shuffle controls.

W3 / Q3. One-sided enhancement adds support rather than checking consistency. Can incidental details promote incompatible contexts, and how meaningful is the gain over independent Fusion (24.02% to 24.20% T2V R@1)?

1. Why the update is one-sided.

ReContext addresses partial relevance, not context-level entailment: unmentioned actions within a relevant activity are not necessarily contradictory. Penalizing unmatched fragments would also conflict with multi-activity descriptions. The enhancement follows:

Hk=Ck+β[Ek−Ck]+.H_k=C_k+\beta[E_k-C_k]_+.

It uses only context kk's own members; CkC_k scores the full description, while the video-wide branch supports other fragments. This is positive, membership-constrained evidence, not a consistency guarantee: incidental matches can cause spurious enhancement, motivating a reliability test.

2. Reliability-gated coupling: when and why membership helps.

We tested a gate that applies coupling only when strong context and member evidence coincide within a structurally reliable context. Let Ck,EkC_k,E_k be context compatibility and its own member support, and let C^k,E^k\widehat C_k,\widehat E_k denote responses normalized by fixed training-split percentiles:

A=max⁡kmin⁡(C^k,E^k),Bk=[Ek−Ck]+,A=\max_k\min(\widehat C_k,\widehat E_k),\quad B_k=[E_k-C_k]_+,

Q=∑kBkqk∑kBk+ϵ,Q=\frac{\sum_k B_kq_k}{\sum_k B_k+\epsilon},

qk=0.30qcov,k+0.20qtemp,k+0.50qvis,k,q_k=0.30q_{\mathrm{cov},k}+0.20q_{\mathrm{temp},k}+0.50q_{\mathrm{vis},k},

qcov,k=min⁡(1,∣Ik∣/3),q_{\mathrm{cov},k}=\min(1,|I_k|/3),

g(V,T)=1[max⁡kBk>0  ∧  0.65A+0.35Q≥0.60],g(V,T)=\mathbf{1}\left[\max_k B_k>0\;\land\;0.65A+0.35Q\ge0.60\right],

Sgate=F+g(V,T)(Uβ−F).S_{\mathrm{gate}}=F+g(V,T)(U_\beta-F).

Here qtemp,kq_{\mathrm{temp},k} measures temporal coherence and qvis,kq_{\mathrm{vis},k} member-feature consistency; each is in [0,1][0,1]. FF is independent Fusion and UβU_\beta the original ReContext score. When structure is unreliable, the gate falls back to Fusion rather than amplifying a spurious match. All gate weights, thresholds, and response percentiles were selected on the training split and frozen for testing. Original features and scoring parameters stay fixed; no extra API calls, encoding, or gradient training is needed.

Full ActivityNet-Event results (R@1, %):

MethodT2VV2T-Avg.
Independent Fusion24.0210.91
Original ReContext24.2011.08
Gated + Shuffle (10 seeds)24.1711.03
Reliability-Gated ReContext24.7011.29

Gating improves over Fusion by +0.68 pp T2V (95% CI [+0.46, +0.90]) and +0.37 pp V2T ([+0.25, +0.51]), on the full evaluation set, not a selected query subset. To test genuine membership fairly, we recompute the gate after every shuffled context–member reassignment; no decisions from original memberships are reused, and all training-derived parameters remain fixed. Genuine pairing beats this gated Shuffle by +0.5320 pp T2V (95% CI [+0.3816, +0.6851]) and +0.2531 pp V2T ([+0.1482, +0.3604]), using unrounded results. Both intervals exclude zero. This supports original memberships within the full gated procedure, not the enhancement term in isolation.

3. Overall framework versus incremental coupling.

The 24.02 → 24.20 comparison isolates unconditional coupling after structured parsing, local/context representations, calibration, and video-wide support are already present. It cannot summarize the framework's contribution. With the same parsed representations, a conventional Plain Fusion achieves 19.66 T2V / 8.81 V2T, versus 24.20 / 11.08 for original ReContext: +4.54 pp T2V (95% CI [3.87, 5.21]) and +2.28 pp V2T ([1.91, 2.64]). The frozen baseline scores 17.41 / 8.30. Within the context-only branch, membership enhancement also increases T2V R@1 from 22.69 to 23.25 (+0.56 pp).

The incremental original ReContext–Fusion difference is +0.18 pp T2V (95% CI [−0.09, +0.46]) and +0.17 pp V2T ([+0.01, +0.32]). Its T2V interval includes zero. Historical Shuffle diagnostics show why: all queries: −0.04 pp [−0.18, 0.10]; aligned (1,758 queries): +1.05 [0.53, 1.58]; aligned with stronger enhancement (442): +2.38 [1.25, 3.66]; and low alignment (264): −1.78 [−3.28, −0.51]. This independently recomputed shuffled control establishes that the preserved context–member associations matter within the gated scoring design. The two branches address complementary aggregation issues: context–member association and cross-activity fragment support.


We welcome further discussion and hope these results support reconsideration of the assessment.

Thank you for the detailed response and the additional experiments.

The revised formulation and the max-only comparison address my concern about the mismatch between the analysis and the implementation. The reference-pool experiment also reassures me that aggregate retrieval performance is stable under the tested subsampling.

The independently recomputed shuffle control is the main reason I am raising my rating. On the full ActivityNet-Event benchmark, genuine memberships improve R@1 by 0.53 percentage points for text-to-video retrieval and 0.25 for video-to-text retrieval, with both confidence intervals excluding zero. This is convincing evidence that the context–member associations contribute to the gated method. It remains unclear how often the gate prevents incidental matches from boosting an incompatible context, but the new evidence is sufficient for me to raise my rating to 3

AuthorsOpenReview ↗

Thanks and Follow-Up on Reliability Gating

Thank you for the constructive discussion and for raising the score!

We appreciated the opportunity to clarify the method and share additional experimental evidence. To follow up on your remaining question, we examined how often the gate suppresses boosts for potential distractors.

It remains unclear how often the gate prevents incidental matches from boosting an incompatible context

In our Top-1 retrieval analysis, we compared ground-truth (GT) and non-GT candidates.

MetricRateCount
GT enhancement retention53.50%1,115 / 2,084
Non-GT enhancement retention22.22%1,316 / 5,922
Non-GT enhancement suppression77.78%4,606 / 5,922

These statistics cover candidates eligible for positive context enhancement before gating. Retention measures the proportion whose enhancement remains active after gating, while suppression measures the proportion whose enhancement is disabled. GT and non-GT denote annotated positive and non-positive retrieval candidates, respectively.

The gate suppresses 77.78% of non-GT enhancement opportunities, while retaining GT enhancement at 2.41× the non-GT rate (53.50% vs. 22.22%). This demonstrates that gating preferentially preserves enhancement for correct candidates while filtering a substantial proportion of potentially misleading support for non-GT candidates.

We use non-GT candidates as a proxy for potential distractors, so their enhancement suppression rate provides indirect evidence that the gate can limit misleading boosts. Coupling uses each context's own members to compute its enhancement, and gating controls whether the enhanced score is used for a given video–text pair.


Thank you again for your careful reading and for the thoughtful feedback you shared throughout the discussion.

If anything still stands in the way of a higher score, we would be grateful if you could point us to it so we can address it before the discussion period closes.

Reviewer_wCT7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ReContext, a training-free framework for multi-event video–text retrieval. The authors argue that independently aggregating local element matches and activity-level context scores discards important local–context associations. To address this limitation, ReContext introduces context-coupled selection, which enhances each activity context using matching scores from its constituent elements, and video-wide aggregation, which collects supporting evidence for different text fragments across the video. Experiments on Charades-Event and ActivityNet-Event demonstrate improvements in both text-to-video and video-to-text retrieval.

Critical strengths

The authors evaluate their method on two benchmarks in both retrieval directions, showing improvements over existing baselines without task-specific training.

Critical weaknesses
  1. The paper uses Qwen3.5-Plus (Qwen3.5-397B-A17B, if correct) for parsing. It's unfair to compare it to other embedding-only baselines with far fewer parameters.

  2. In Table 4, ReContext improves over independent branch fusion by only 0.18 points in T2V R@1. More importantly, on the local-confusion subset in Table 5, which directly targets the paper's motivation, the improvement over Fusion is only 0.10 points. These findings raise questions about whether the proposed coupling mechanism, rather than simply incorporating contextual information, is responsible for the observed improvements.

Questions for authors
  1. How were the hyperparameters selected, and was a separate validation set used? While Appendix provides several sensitivity analyses, it does not examine the low calibration temperatures. How robust are the results to these temperatures and to changes in the reference pool?

  2. Appendix reports component runtimes, but could the authors also clarify end-to-end query latency and API costs? Given the additional Qwen3.5-Plus parser, comparisons with InternVideo2 alone should account for the extra resources and distinguish gains from semantic preprocessing from those of the proposed scoring mechanism.

  3. Does ReContext consistently improve retrieval with a more recent embedding backbone, such as Qwen3-VL-Embedding? Comparing the new backbone alone against its ReContext-enhanced counterpart would help establish whether the gains generalize beyond InternVideo2. It's also valuable to provide a strong embedding baseline for the benchmark.

  4. Semantic Structure Construction remains underspecified. Could the authors provide the prompts, frame-sampling strategy, temporal localization procedure, and context-grouping details? Beyond the schematic example in Figure 2, an assessment of parsing accuracy and failure cases would help, particularly for longer videos under limited frame/token budgets.

6 comments

We thank the reviewer for examining fairness, coupling, and calibration. We provide matched controls, gated results, and robustness tests.

W1. Are comparisons with smaller baselines unfair because ReContext uses Qwen3.5-Plus for parsing?

ReContext is a training-free scoring framework, not a matched-size encoder. Qwen3.5-Plus parses reusable structures; frozen InternVideo2 encodes them. Embedding-only comparisons are resource-asymmetric. Plain Fusion shares parsing and embeddings, but uses basic fusion:

ActivityNet-Event R@1 (%)T2VV2T-Avg.
InternVideo2 (flat)17.418.30
Plain Fusion (same parsing)19.668.81
ReContext24.2011.08
ReContext − Plain Fusion (95% CI, pp)+4.54 [3.87, 5.21]+2.28 [1.91, 2.64]

The +4.54/+2.28 pp beyond Plain Fusion comes from scoring (calibration, aggregation, coupling), not extra parsing/API calls.

W2. Coupling adds only +0.18 pp T2V R@1 over Fusion; the local-confusion subset shows +0.10 pp, and Shuffle shows no overall pairing advantage. Does membership matter?

1. Reliability-gated coupling.

Fusion shares ReContext’s parsing, calibration, and fusion; only coupling differs. Unconditional coupling adds +0.1844 pp T2V [−0.0912, 0.4623] and +0.1704 pp V2T-Avg. [0.0144, 0.3231] over Fusion (95% CIs), but −0.04 pp [−0.18, 0.10] versus Shuffle overall. We therefore test when genuine pairing helps.

We activate coupling only when context–member evidence aligns and membership is reliable. Let CkC_k be context compatibility, EkE_k member support, and tildes fixed training-percentile normalization:

A=max⁡kmin⁡(C~k,E~k),Bk=[Ek−Ck]+,Q=∑kBkqk∑kBk+ϵ,A=\max_k\min(\widetilde C_k,\widetilde E_k),\quad B_k=[E_k-C_k]_+,\quad Q=\frac{\sum_kB_kq_k}{\sum_kB_k+\epsilon},

qk=0.30qcov,k+0.20qtemp,k+0.50qvis,k,qcov,k=min⁡(1,∣Ik∣/3).q_k=0.30q_{\rm cov,k}+0.20q_{\rm temp,k}+0.50q_{\rm vis,k},\quad q_{\rm cov,k}=\min(1,|I_k|/3).

qtempq_{\rm temp} and qvisq_{\rm vis} assess temporal and feature consistency in [0,1][0,1]. The rule is

g=1[max⁡kBk>0 ∧ 0.65A+0.35Q≥0.60],Sgate=F+g(Uβ−F),g=\mathbf1[\max_kB_k>0\ \land\ 0.65A+0.35Q\ge0.60],\quad S_{\rm gate}=F+g(U_\beta-F),

Here FF is Fusion; UβU_\beta is original ReContext. All gate settings were selected on training data and frozen for testing. Features and original scoring stay fixed; no extra API calls, encoder passes, or gradient training.

Full ActivityNet-Event benchmarkT2V R@1V2T R@1-Avg.
Fusion24.017310.9126
Original ReContext24.201711.0829
Gated + Shuffle (10 seeds)24.165411.0332
Reliability-gated ReContext24.697411.2862

Over all queries, gated coupling exceeds Fusion by +0.68 pp T2V [0.46, 0.90] and +0.37 pp V2T [0.25, 0.51] (95% CIs). After each shuffle we recompute the gate from shuffled assignments, keeping parameters fixed, never reusing original gate decisions. Genuine pairing then exceeds gated Shuffle by +0.5320 pp T2V [0.3816, 0.6851] and +0.2531 pp V2T [0.1482, 0.3604] (95% CIs; unrounded). Both CIs exclude zero, isolating genuine pairing.

2. Framework versus incremental coupling.

The 1,981-query local-confusion subset tests context, not membership reliability: Local/Fusion/ReContext obtain 19.08/29.63/29.73 T2V R@1; Fusion provides +10.55 pp over Local. Original Shuffle controls show +1.05 pp [0.53, 1.58] on 1,758 aligned queries and +2.38 pp [1.25, 3.66] on 442 with stronger enhancement (Appendix D.3). Low-alignment queries lose −1.78 pp [−3.28, −0.51]; framework gains remain distinct.

Q1. How were hyperparameters selected? Was there a separate validation set? Are low temperatures and reference pools robust?

1. Calibration/selection.

Eq. (6) normalizes each text component against references, without assuming comparable element/context scales. We chose τe=0.0125\tau_e=0.0125, τc=0.015\tau_c=0.015, β=0.6\beta=0.6, α=0.15\alpha=0.15, λ=0.8\lambda=0.8 on 100 ActivityNet training videos and fixed them across benchmarks; this was not an independent validation set.

2. Reference-pool robustness (three seeds, no retuning).

PoolT2V R@1 (%)V2T R@1-Avg. (%)
100%24.201711.0829
75%24.1825 ± 0.012011.0749 ± 0.0097
50%24.1748 ± 0.008811.0684 ± 0.0079

At 50% the changes are −0.027/−0.015 pp T2V/V2T, supporting tested pool sizes, not untested temperatures.

We further report query timings, a fast T2V parsing control, backbone results, and parsing checks.

Q2. What are end-to-end query latency and API costs, and how much gain comes from semantic preprocessing rather than scoring?

1. Online cost.

Gallery videos are parsed and encoded once, not per query–candidate pair. ActivityNet-Event timings:

StageT2V (4,877 videos)V2T (17,497 texts)
Uncached query API parsing7.23 s (text)8.71 s (video)
Uncached query encoding0.030 s1.270 s
Cached metadata load/conversion916.74 ms1,932.48 ms
Cached feature load/preparation506.19 ms420.62 ms
Score, rank, return11.82 ms2.74 ms
Total cached retrieval (incl. gallery loading)1.435 s2.356 s

Including API parsing and encoding, an uncached query takes ~8.70 s T2V / ~12.34 s V2T by component sum; cache policy affects latency.

2. Fast T2V text splitting.

An action-based sentence-splitting script needs 130.21 ms/query versus 7.23 s API parsing (~55.5× faster), producing identical element splits for 98.2% of evaluated texts relative to the API. Replacing the API time in the cached-gallery breakdown gives ~1.60 s/query, including gallery loading and 30 ms encoding. This is a component-wise estimate, not an end-to-end measurement; splitting agreement does not imply identical retrieval or semantic annotations.

3. API cost and attribution.

Historical parsing costs around $0.0018/video ($9/gallery) and $0.00045/text query; pricing and token counts vary. Cached queries add no API cost. Fusion and ReContext share preprocessing/embeddings: their +0.18/+0.17 pp T2V/V2T difference comes from scoring, without extra API or encoder calls; Plain Fusion controls broader scoring gains (W1).

Q3. Does the method transfer to another backbone, including a newer embedding model such as Qwen3-VL-Embedding?

The framework needs a shared embedding space, not InternVideo2-specific layers. We tested frozen CLIP ViT-L/14 with matched preprocessing within each pair:

ActivityNet-Event R@K (%)R@1R@5R@10R@50
T2V: CLIP1.55915.59149.623725.1613
T2V: CLIP + ReContext2.20436.774211.505430.0806
V2T-Avg.: CLIP1.03883.08934.735413.4762
V2T-Avg.: CLIP + ReContext1.36173.61505.278715.6631

All cutoffs improve (+0.6452/+0.3228 pp T2V/V2T R@1), supporting two frozen encoder families, not untested Qwen3-VL-Embedding.

Q4. What prompts, frame sampling, temporal localization, context grouping, and parsing-quality checks are used, especially for long videos?

1. Video prompt/localization.

Qwen3.5-Plus extracts timestamped atomic actions, objects, and interactions, grouping them fine-to-coarse by activity. Groups retain IDs, times, and rationales; memberships may overlap or bridge gaps. Parsing samples 2 FPS, at most 8,000 frames; timestamps are converted to seconds, contexts follow member spans. Embedding uses up to four frames per element/context, without ground-truth temporal boundaries.

2. Text prompt.

Remove subjective/camera narration, retain observable facts, extract atomic actions and temporal/causal relations, and rebuild standalone fragments with actors, objects, and modifiers. The 130.21 ms script (Q2) replaces only text element splitting.

3. Human evaluation.

Two annotators independently assessed 100 ActivityNet-Event video–text pairs; a third adjudicated disagreements. Pooled Cohen's κ=0.92\kappa=0.92.

DimensionValid / totalValidity
Text semantic fidelity95/10095.0%
Visual grounding461/47197.9%
Context coherence253/27891.0%
Context–member association442/47493.2%

We welcome further questions and discussion and hope the evidence supports reconsideration of the rating.

I thank the authors for the additional experiments. However, I maintain my score for the reasons below:

  1. Using Qwen3.5-Plus for parsing makes comparisons with embedding-only baselines resource-asymmetric, while the high query latency limits practicality.
  2. The central coupling mechanism provides only marginal gains over independent fusion. The gated variant offers additional evidence, but the improvements remain modest.
  3. Strong recent embedding baselines, such as Qwen3-VL-Embedding, are still missing. Demonstrating improvements over such models would better justify the system’s tradeoff between performance and additional complexity.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response by Authors

We thank all four reviewers for their constructive feedback. We have revised the manuscript to clarify the scoring formulation and theoretical analysis, and provided additional experiments on membership coupling, calibration robustness, parsing reliability, retrieval efficiency, and backbone generalization.

These results further support ReContext's central design: coupling local evidence with its corresponding activity context before aggregation, while retaining video-wide support for descriptions spanning multiple activities. In particular, our new reliability-gated experiments provide stronger evidence for the contribution of genuine context–member associations.

Several related concerns were raised across reviews, particularly regarding coupling and scoring consistency (hPrp, wCT7, MsqJ), calibration (hPrp, wCT7, MsqJ), and parsing, efficiency, and backbone comparisons (hPrp, Eirb, wCT7).

Main revisions and additional evidence:

  • Reliability-aware membership coupling. We introduced a training-selected reliability gate that activates coupling when context and member evidence align within a structurally reliable context. On the full ActivityNet-Event benchmark, gated ReContext achieves 24.70% T2V / 11.29% V2T R@1, improving over Independent Fusion by +0.68 / +0.37. Crucially, genuine memberships outperform independently recomputed Gated Shuffle by +0.53 / +0.25 (95% CIs: [0.38, 0.69] / [0.15, 0.36]), directly supporting the value of preserving context–member associations.

  • Controlled scoring contributions. Using identical parsing and frozen representations, ActivityNet-Event T2V/V2T R@1 improves from 19.66/8.81 (Plain Fusion) to 24.02/10.91 (calibrated Independent Fusion) and 24.20/11.08 (original ReContext). These controlled comparisons distinguish improvements from structured calibration and aggregation from those of semantic preprocessing, while isolating the contribution of coupling.

  • Consistent scoring formulation and theoretical analysis. The original max-only formulation is a mathematically valid special case that captures the core principle of coupling local evidence with context. Following the reviewer's suggestion, we provide a direct comparison between max-only and Top-3 aggregation, showing modest R@1 improvements with Top-3. We therefore adopt Top-3 in the revised Fig. 2 and Eqs. (7)–(10) (Sec. 4) and Appendix C to reduce reliance on extreme responses and ensure consistency with the implementation.

  • Calibration robustness. We clarified reference-based calibration and parameter selection using 100 training-split videos, with parameters fixed across benchmark evaluations. Reducing the calibration reference pool to 50% changes T2V/V2T R@1 by just 0.027/0.015 points, respectively, without retuning.

  • Parsing reliability and practical efficiency. We detailed visual and linguistic parsing procedures. Human evaluation reports 95.0% textual fidelity, 97.9% visual grounding, 91.0% context coherence, and 93.2% membership validity (Cohen's κ = 0.92). For online T2V, a lightweight text splitter takes 130.21 ms/query, reproducing API element segmentation on 98.2% of evaluated descriptions. Scoring and ranking over the 4,877-video gallery takes 11.82 ms/query, with coupling requiring no additional API calls or encoder passes.

  • Backbone generalization. With frozen CLIP ViT-L/14, ReContext improves T2V R@1 from 1.56% to 2.20% and V2T R@1 from 1.04% to 1.36%, with gains across all reported recall cutoffs, supporting the transferability of the scoring framework.

Together, these additional analyses and revisions provide a clearer characterization of ReContext's scoring mechanism, the contribution of genuine membership associations, and its robustness and practical applicability.

We appreciate the reviewers' feedback and hope these results support a more complete assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.