- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce VEDJE, a video reranker that compresses each sampled frame into a few cached tokens offline. They evaluate on MSR-VTT, MSVD, DiDeMo, and ActivityNet across several first-stage retrievers.
Critical strengths
- This paper is a well-executed extension of EDJE to video, taking EDJE’s cached joint-reranking recipe from images to video and shows that it transfers.
Critical weaknesses
W1. Missing comparisons and related work undermine the paper’s novelty and central claims.
The paper’s contributions are a compact cache read by a lightweight joint reranker, feature-change supervision, and “scalable” retrieval. Each is evaluated without comparison to the work that most directly addresses it. That leaves both the novelty and the main claims unsubstantiated.
Novelty
The paper itself notes that EDJE already outlines the video extension via temporal compression and positional encodings, and VEDJE largely implements that outline. Beyond EDJE, each remaining component has close precedent that goes uncited or compared against:
- Caching document representations offline and running a small interaction model online, with a compressed or quantized cache. This is the core of PreTTR [4], MORES [5], Poly-encoders [6], and SDR [7].
- Query-agnostic, fixed-budget compression of video for retrieval. This is studied by AGC [1], H-Pool [2], and TempMe [3]. AGC includes memory-token compressors closely resembling VEDJE’s learned queries.
- The residual first-stage prior. This mirrors score interpolation [9] and score injection into a MiniLM cross-encoder [8]. The prior carries much of the result: the joint stream alone reaches 43.8 T2V R@1, below the 50.1 first stage (Table 11).
Comparisons.
No existing method is run on the paper’s candidates. Video-native rerankers such as Qwen3-VL-Reranker [12] and RankVideo [13] are absent, even though the motivation rests on their query-time cost, which is asserted but never measured. Also absent are compressed-representation alternatives: AGC [1] (56.9 MSR-VTT R@1 at 32 tokens per video), training-free H-Pool [2] on the same VideoPrism features, and embedding-based rerankers such as PE-Rank [14] and E2Rank [15]. The pooling baselines in Table 4 are constructed by the authors rather than drawn from this literature.
Scale.
Despite “scalable” in the title, all evaluations use corpora of at most 5K videos. Large corpora produce denser near-duplicate candidate lists, which is exactly where a query-blind compressed cache could lose discriminative evidence. MultiVENT 2.0 [16] (110K-video test collection, already used by AGC [1]) would directly test this.
Without these comparisons, a reader cannot determine three things: whether the gains come from the video-specific design or from known ingredients (e.g., the first-stage prior), whether VEDJE improves on existing points of the accuracy–cost frontier, and whether the results hold at the scale the title claims. Addressing this would require same-candidate comparisons against AGC/H-Pool and at least one video-native reranker with measured latency. It would also need an evaluation on a large-scale benchmark such as MultiVENT 2.0. Without them, the paper is better framed as a storage- and latency-efficient empirical study than as introducing new methods for scalable retrieval.
[1] https://dl.acm.org/doi/10.1145/3805712.3809589 [SIGIR 2026] [2] https://arxiv.org/abs/2409.14683 [3] https://arxiv.org/abs/2409.01156 [ICLR 2025] [4] https://dl.acm.org/doi/10.1145/3397271.3401093 [SIGIR 2020] [5] https://aclanthology.org/2020.emnlp-main.342/ [EMNLP 2020] [6] https://arxiv.org/abs/1905.01969 [ICLR 2020] [7] https://aclanthology.org/2022.acl-long.457/ [ACL 2022] [8] https://dl.acm.org/doi/10.1007/978-3-031-28244-7_5 [ECIR 2023] [9] https://dl.acm.org/doi/10.1145/3471158.3472233 [ICTIR 2021] [12] https://arxiv.org/abs/2601.04720 [13] https://arxiv.org/abs/2602.02444 [14] https://dl.acm.org/doi/10.1145/3696410.3714554 [WWW 2025] [15] https://arxiv.org/abs/2510.22733 [16] https://arxiv.org/abs/2410.11619 [CVPR 2025]
W2. The efficiency comparison doesn’t reflect query cost.
The paper’s central selling point is efficiency, but the evidence it uses (online parameter counts in Figure 1) doesn’t measure what it’s used to claim. It has the following issues:
- Single-stage retrievers (e.g., Video-ColBERT) run their text encoder once per query against precomputed video representations. VEDJE runs its cross-encoder once per candidate (20 forward passes per query, over up to 64 text and 64 cache tokens each). Its per-query compute therefore grows with the number of candidates, which a parameter count hides.
- Single-stage retrievers are plotted alongside two-stage systems, implying an apples-to-apples comparison that doesn’t exist.
- In the default configuration, the cache is built with a different backbone than the first stage (VideoPrism-Base vs. VideoPrism-LvT-B, Table 7). This requires a second visual encoding pass over the corpus and a second per-video store alongside the first-stage index. That cost matters most at the scale the paper claims to operate at.
Efficiency is better measured with per-query latency or FLOPs on the same hardware and candidate count. For a figure like Figure 1, it would be better to plot latency, or at least to separate first-stage retrievers from rerankers. Indexing cost and storage are also missing, including both the first-stage index and the cache. This is consequential because VEDJE trails CaRe-DPO and only ties LamRA in accuracy (Table 1a), so its case rests on cost, and the evidence presented can’t support that case.
Questions for authors
none, see above.
5 comments
W1. Role of the first-stage score. The prior is an important ingredient: Table 11’s independently trained joint-only model reaches 43.8 T2V R@1, compared with 50.1 for the first stage. Our new controls measure what the complete training procedure adds beyond combining independently learned scores.
Taking the stronger control in each direction (ordinary interpolation for T2V and a learned fusion head for V2T) gives 52.4/51.5, compared with 54.6/54.6 for VEDJE. The respective gains are +2.2 (95% CI [+1.0, +3.4]) and +3.1 ([+2.2, +4.0]).
We also freeze the learned caches and train fresh readers using the same protocol, with access to the first-stage score:
| How the frozen cache was trained | T2V | V2T |
|---|---|---|
| Without the first-stage prior | 52.5 | 52.1 |
| Jointly with VEDJE’s prior-aware scoring head | 54.5 | 54.5 |
The 2.0/2.4-point advantage remains after retraining the reader, showing that cache training contributes beyond adding the prior afterward. Architecturally, the embedded prior enters the scoring head after joint text–cache encoding; cache construction remains query-independent (Eq. 5).
Relevant precedents include score interpolation [9], supplying the BM25 score as text to a trained cross-encoder [8], and CLEAR (Gao et al., 2021), which trains dense text representations to complement BM25. Our frozen-cache experiment tests the value of learning the video cache jointly with the prior-aware scorer.
W1. Same-candidate accuracy–cost comparison. AGC-32 (released MSR-VTT checkpoint) and Qwen3-VL-Reranker-2B [12] rerank the same fine-tuned VideoCLIP-XL top-20 candidates as VEDJE. T2V timings use one NVIDIA L40S, with one query at a time.
| Method | T2V R@1 | Query total (ms) | Rerank only (ms) | GPU memory (GB) |
|---|---|---|---|---|
| First stage | 56.2 | 5.71 | — | 2.39 |
| AGC-32 | 56.5 | 14.98 | 9.27 | 10.01 |
| AGC-32 + score | 58.1 | 14.97 | 9.26 | 10.02 |
| Qwen3-VL-Reranker-2B | 53.1 | 747.39 | 741.68 | 6.64 |
| Qwen3-VL-Reranker-2B + score | 57.3 | 753.84 | 748.13 | 6.65 |
| VEDJE | 59.8 | 9.66 | 3.95 | 2.48 |
Against the validation-selected fusion variants, VEDJE gains 1.7 R@1 points over AGC (95% CI [+0.2, +3.2]) and 2.5 over Qwen ([+0.5, +4.5]). The query totals give 1.55× lower latency than fused AGC and approximately 78× lower latency than fused Qwen. VEDJE uses approximately one quarter of fused AGC’s measured GPU memory. As a fusion sensitivity check, fixed 50/50 weights give AGC and Qwen 58.7 and 58.0 R@1, respectively, also below VEDJE’s 59.8.
On VideoPrism candidates, fused Qwen reaches higher accuracy (57.9 versus 54.6) at higher latency (731 versus 5.79 ms); that fusion weight is transferred from the VideoCLIP-XL validation experiment. These results characterize the accuracy–cost tradeoff under both candidate generators.
PE-Rank [14] replaces passage text with projected embeddings for LLM-based listwise ranking. E2Rank [15] encodes the query together with candidate passage text, then scores cached document embeddings by cosine similarity. Both provide related text-retrieval approaches to efficient reranking with reusable representations.
All reported intervals use paired query bootstrapping, averaging each query’s metric contribution across training seeds where three seeds were evaluated.
Additional reference: Gao et al., Complement Lexical Retrieval Model with Semantic Residual Embeddings, ECIR 2021.
W1. Evaluation at scale. We evaluate T2V retrieval on the full 109,800-video MultiVENT 2.0 test gallery, using a frozen PE-Core-B first stage. All rerankers, including the reader in our H-Pool adaptation, are trained on MultiVENT training queries and rerank identical top-100 candidate lists. Video inputs are the same 16 sampled frames, with no audio or transcripts at inference. Timings use one NVIDIA L40S.
| Method | nDCG@10 | R@10 | Complete query (ms) |
|---|---|---|---|
| First stage | 31.0 | 33.0 | 5.0 |
| H-Pool-64 with our joint reader | 32.5 | 34.2 | 8.9 |
| VEDJE-16 | 36.0 | 36.6 | 6.8 |
| VEDJE-64 | 37.5 | 38.0 | 8.5 |
VEDJE-64 improves nDCG@10 by 6.5 points over the first stage and 5.0 over H-Pool at the same cache budget. VEDJE-16 retains a 5.0-point improvement over the first stage with one quarter of the cache payload. On the 20% of queries with the smallest first-stage top-1/top-2 score margins, VEDJE-64 gains 10.5 points (95% CI [+8.8, +12.2]).
Published MultiVENT results provide additional context: AGC reports 46.3 nDCG@10 with an audiovisual backbone; RankVideo reports 47.8 with a CLIP first stage; and Video-ColBERT reports 42.2 (Table 2 of [13]). These use different configurations. Our controlled comparison measures VEDJE’s added value with the first stage, visual inputs and candidate lists fixed; our direct latency comparisons use the locally profiled systems.
W2. Query cost and single-stage comparisons. VEDJE scores candidates in batches, and its computation grows with the candidate count. With the default 64-token VideoPrism cache, complete T2V query latency is 5.79/6.21/7.83 ms for 20/50/100 candidates (Table 6).
On the same NVIDIA L40S and MSR-VTT queries and gallery, native single-stage AGC-32 achieves 56.4 T2V R@1 in 8.79 ms using 7.77 GB of GPU memory. VEDJE with fine-tuned VideoCLIP-XL achieves 59.8 R@1 using 2.48 GB, with its measured stages summing to 9.66 ms: 3.4 additional R@1 points for 0.87 ms, with approximately one third of AGC’s GPU memory use.
We will replace Figure 1’s parameter-count comparison with T2V R@1 versus query latency, using the same NVIDIA L40S and MSR-VTT queries and gallery. Separate panels will show complete retrieval pipelines, with native single-stage and two-stage systems clearly marked, and rerankers evaluated on identical VideoCLIP-XL top-20 candidates. Each point will identify its first stage and candidate count; the caption will specify which totals are calculated from measured stage costs. LamRA and CaRe-DPO will remain published accuracy references in Table 1a; measured efficiency claims will be limited to the systems we profiled.
W2. Offline construction and storage. The default configuration uses VideoPrism-LvT-B for first-stage retrieval and VideoPrism-Base for cache features. Complete construction takes 88 ms/video, versus 62 ms for the first-stage representation alone: 26 ms additional offline cost.
The measurement includes decoding and preprocessing 16 frames at 288×288, both visual encoders, compression and writing, on one NVIDIA L40S at batch size 1. The visual encoders use FP32 and the compressor uses BF16.
For this default configuration, packed representations occupy 15,370 bytes/video at 16 tokens or 52,234 bytes/video at 64 tokens, including the first-stage vector and identifiers. ANN infrastructure and filesystem overhead are excluded. The 64-token cache payload for the full MultiVENT gallery is 5.4 GB.
Our scalability claim concerns fixed cache storage per video and bounded neural reranking work at a fixed candidate count. First-stage search and total index storage remain dependent on gallery size.
We will incorporate all additional results, protocols and explanations into the revised manuscript and appendix, including expanded Related Work and the revised Figure 1. Thank you for prompting these additions; we hope the new evidence addresses your concerns and would be grateful for your reassessment. We would be happy to respond to any remaining concerns.
Thank you for identifying the comparisons needed to assess the contribution. We have added same-candidate AGC and H-Pool comparisons, measured Qwen3-VL-Reranker on the same hardware, and evaluated the full 109,800-video MultiVENT 2.0 gallery.
W1. Contribution and relation to prior work. Our goal is accurate joint video–text matching with a small cache and low query cost. VEDJE compresses each sampled frame’s features into a separate token group, trains the cache with feature-change supervision, and uses a small joint reader to combine those groups with the query.
EDJE is the closest architectural precedent: it already combines a compressed visual cache with a lightweight joint reader, and its Appendix I outlines video extensions using temporal aggregation. VEDJE builds on this design and studies how to organize and train the video cache.
The text-retrieval precedents are also relevant. PreTTR [4] and MORES [5] precompute document-token representations and defer query–document interaction. PreTTR and SDR [7] additionally compress the stored token representations. Poly-encoders [6] cache one vector per candidate and use it to attend over multiple context summaries. These works establish precomputation and subsequent interaction; our methodological contribution concerns the structure and training of a compact video cache.
Preserving frame groups. We compare against an EDJE-inspired global resampler that aggregates features across all 16 frames, with frame-position encodings and the same compressor architecture and approximately 5M parameters as VEDJE. These controls isolate the value of frame-group organization and feature-change training while keeping the first-stage prior and joint reader fixed. All rows below use the same VideoPrism features, top-20 candidates, first-stage score, joint reader and 64-token/48-KiB budget. Results are MSR-VTT R@1.
Preserving frame groups adds 2.0 points in both directions over the capacity-matched global resampler before feature-change supervision is introduced. Full VEDJE exceeds H-Pool by 1.4/1.2 points, with 95% paired confidence intervals [+0.8, +2.0]/[+0.5, +1.9]. These comparisons hold the first-stage prior fixed.
AGC, H-Pool and TempMe. AGC [1] uses attention-guided centroid selection and weighted clustering; its paper evaluates learned memory-token compression as a separate baseline. H-Pool [2] uses hierarchical clustering and pooling without additional training. Both were developed for compact representations used in MaxSim matching. Our H-Pool adaptation applies the published pooling algorithm to our frozen features and trains a projection and joint reader. We separately evaluate the released AGC checkpoint below.
On the same VEDJE cache, joint reading also improves over trained MaxSim: 54.6 versus 51.9 T2V R@1 (Table 5). TempMe [3] reduces video-encoding computation by progressively merging spatial and temporal tokens during encoding; VEDJE compresses extracted features for reusable query-time matching.
Feature-change supervision. At 64 tokens, predicting feature changes reaches 54.6/54.6, versus 54.2/53.9 for current-frame reconstruction and 52.8/53.8 for future-feature prediction (Table 18). For T2V, the benefit is greater under tighter compression: reducing 64 to 16 tokens costs 1.5 points without this supervision, but only 0.2 with it. The prediction head is discarded after training, adding no query-time computation.