acceptodds
Under review as a conference paper at ICLR 2027

FlashRank: Fast Listwise Video Reranking from Diverse Compressed Representations

Abstract

Video reranking aims to improve the quality of candidate list returned by an upstream retrieval system, however, existing rerankers are limited by their latency, because of requiring re-encoding sampled frames and audio at query time. We propose FlashRank, a listwise reranker that reranks candidate vidoes based on contexutalized representations from a multimodal video index instead of raw videos. FlashRank projects cached video, audio, and text embeddings into an LLM, which jointly contextualizes the candidate list and combines modality-specific evidence for relevance estimation. We jointly adapt the query encoder with reranking objective while keeping all cached video representations fixed, enabling improvements to both first-stage retrieval and reranking without rebuilding the index. Across nine video retrieval datasets, FlashRank achieves a speedup over typical video-based rerankers with comparable, or even greater, nDCG@10. We further show that large scale text ranking supervision transfers effectively to video reranking, improving relevance estimation from compressed multimodal representations. Together, this makes effective video reranking practical for deployment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Listwise reranking, Compression embedding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_chL1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlashRank, a listwise video reranking framework that operates on precomputed multimodal representations instead of re-encoding raw videos at query time. The method projects cached video, audio, and text embeddings into a causal LLM, contextualizes multiple candidates jointly, and computes relevance scores from contextualized hidden states combined with residual embeddings and first-stage retrieval similarities. Training consists of representation-to-text alignment, ranking distillation from video relevance scores, auxiliary text-domain ranking supervision, and optional query-encoder adaptation while keeping indexed video representations fixed.

Across nine video retrieval benchmarks, FlashRank-VAT achieves an average nDCG@10 of 0.631 at 0.356 seconds per query, compared with 0.637 at 26.8 seconds for Qwen3-VL-Reranker-8B. Adding offline-generated textual claims increases average nDCG@10 to 0.649 at 7.53 seconds per query. The authors further study modality separation, training objectives, query adaptation, and inference-window configurations.

Critical strengths
  1. Practical efficiency gains from reusing cached multimodal representations.

The strongest aspect of this work is its demonstration that existing video-index representations can support meaningful second-stage ranking without query-time raw-video processing. The reported approximately 74× latency reduction relative to RankVideo, with comparable aggregate retrieval effectiveness, is practically significant. Keeping the document representations fixed also offers deployment advantages when recomputing embeddings for a large video collection is expensive.

  1. Useful empirical investigation of modality-specific information for video retrieval.

The evaluation across nine datasets is reasonably broad, and the comparison between concatenated and separate modality representations provides evidence that preserving video, audio, and text embeddings individually improves retrieval effectiveness. The experiments also demonstrate that text-domain ranking supervision can improve multimodal reranking. These findings have practical value independently of the novelty of the underlying ranking architecture.

Critical weaknesses
  1. Substantial methodological overlap with prior work, with insufficient evidence of a distinct technical contribution.

My primary concern is the limited methodological novelty relative to existing compressed-representation listwise rerankers, particularly Efficient Listwise Reranking with Compressed Document Representations (Déjean and Clinchant, 2026; arXiv:2604.26483) and ResRank (Ke et al., 2026; arXiv:2604.22180).

The overlap extends well beyond the high-level idea of efficient reranking:

Compressed representations as LLM inputs: RRK already replaces full documents with offline-cached continuous representations for listwise reranking.

Nearly identical input structure: RRK (Section 3) uses a query before and after the candidate representations. ResRank (Equation 2) additionally uses an instruction prefix and final EOS token, closely matching FlashRank's Equation 2.

Identical underlying contextual scoring principle: RRK scores candidates using cosine similarity between the final query representation and contextualized candidate states. ResRank (Equations 4–5) further introduces residual addition of the original embedding to the contextualized candidate representation followed by cosine-similarity scoring, closely matching the first term of FlashRank's Equation 4.

Decoding-free listwise inference: Both RRK and ResRank already avoid autoregressive permutation generation and derive candidate relevance scores directly from LLM hidden states.

The overlap is particularly striking with ResRank, which already combines compressed single-token representations, causal listwise contextualization, a repeated-query input structure, residual embeddings, and cosine-similarity scoring. FlashRank's central architecture is therefore largely established by prior work.

I acknowledge that FlashRank introduces several distinctions: modality-specific representations, an additional retrieval-similarity term, log-sum-exp fusion across modalities, and adaptation of the query encoder while preserving the indexed document representations. However, these modifications primarily extend existing ranking formulations rather than introduce a clearly new ranking mechanism. The training objectives similarly build on established alignment, RankNet/ListNet distillation, and retrieval-margin objectives.

The authors cite the relevant prior work, but the methodological discussion does not sufficiently establish why the proposed architecture should be considered a substantive advance beyond adapting these existing methods to multimodal video embeddings.

Most importantly, neither RRK nor ResRank is evaluated as an architecture-matched baseline using the same cached representations. Without such a comparison, the paper does not demonstrate that its proposed scoring and contextualization mechanisms improve meaningfully over a relatively direct adaptation of existing methods.

The distinction of keeping the video index fixed is practically relevant, but that alone does not establish novelty of the reranking algorithm. Given the extent of architectural overlap, I find the demonstrated methodological differentiation insufficient for acceptance.

  1. Experimental comparisons do not adequately isolate the proposed method's effectiveness, and important performance claims depend on unequal input information.

The experiments establish that compressed multimodal representations can yield competitive retrieval performance, but they do not sufficiently demonstrate that FlashRank's particular listwise architecture is responsible for this result.

First, an essential baseline is missing: a simple learned fusion of the same cached video, audio, and text similarities. Such a model could use modality-weighted similarity scores, a small MLP, or a trained pointwise ranking head, without requiring a 7B-parameter LLM. Since the principal gains appear to come from preserving complementary modality information, this comparison is necessary to distinguish the contribution of the representations from the contribution of listwise contextualization.

Table 4 does not resolve this concern. Its pointwise results are obtained by changing the inference configuration of the listwise-trained model, rather than comparing against an independently trained pointwise model with the same inputs and supervision. Consequently, the measured listwise advantage does not establish superiority over a properly optimized pointwise alternative.

Second, Table 1 reveals substantial differences across datasets that the average nDCG@10 obscures. FlashRank-VAT achieves 0.631 versus 0.637 for Qwen3-VL-Reranker-8B, and underperforms this baseline on six of the nine datasets. In particular:

DiDeMo: 0.586 versus 0.765.

ActivityNet: 0.629 versus 0.805.

VATEX: 0.809 versus 0.867.

Conversely, the largest advantages occur on audio-focused benchmarks, including AudioCaps (0.492 versus 0.312) and VGGSound-a (0.450 versus 0.226). However, FlashRank has access to dedicated audio representations, while the evaluated raw-video baselines receive sampled visual frames without equivalent audio evidence. The resulting comparison does not isolate improved ranking capability from differences in available information.

Finally, the strongest efficiency and effectiveness results arise from different configurations. FlashRank-VAT delivers the approximately 74× speedup but slightly underperforms the strongest raw-video baseline in average effectiveness. FlashRank-Claims256 improves average nDCG@10 to 0.649, but increases reranking latency to 7.53 seconds and additionally requires approximately 3–5 seconds of offline preprocessing per video. The resulting improvement remains useful, but is not evidence of simultaneously achieving the claimed maximum speedup and highest effectiveness.

Overall, the reported results support the utility of cached multimodal representations, but provide insufficient evidence for the additional value of the proposed architectural choices over simpler or more closely matched alternatives.

Questions for authors
  1. Differentiation from RRK and ResRank.

Given the close correspondence between FlashRank's Equations 2–4, RRK's Section 3 formulation, and ResRank's Equations 2–5, what is the precise new ranking mechanism introduced by FlashRank beyond modality-specific representation handling and retrieval-score fusion?

Could the authors compare against RRK-style and ResRank-style rerankers adapted to consume the same frozen OmniEmbed representations, with matched training supervision and backbone size? Demonstrating a meaningful improvement under this setting would materially strengthen the novelty claim.

  1. Necessity of LLM-based listwise contextualization.

How does FlashRank compare against a learned weighted combination of the three modality similarities, a small MLP fusion model, and a separately trained pointwise reranker using exactly the same cached representations?

The existing concat-versus-VAT comparison demonstrates the benefit of a particular representation/scoring configuration, but does not establish that listwise LLM contextualization is necessary. This comparison is particularly important given the substantial additional model complexity.

3 comments

1. We thank the reviewer for requesting a closer comparison with RRK and ResRank.

Architectural distinction. Unlike RRK, which compresses text documents into multiple embedding tokens, and ResRank, which jointly learns single-embedding representations of text passages, FlashRank directly reuses modality-specific video, audio, and text embeddings already stored in a frozen multimodal retrieval index for listwise reranking, without video re-encoding or index reconstruction.

Reproducibility and related work. To our knowledge, neither RRK nor ResRank has released the experimental code needed for reproduction, and their representation-construction and training pipelines differ from ours. The ICLR 2027 Reviewer Guidelines explicitly state that comparisons with contemporaneous or arXiv-only work are not required, and that the absence of such comparisons alone cannot justify rejection. Nevertheless, to directly address the reviewer's concern, we implemented RRK-style and ResRank-style adaptations using the same frozen index, top-100 candidate pools, 7B backbone, and ranking supervision (Table R6).

Findings (Table R6). FlashRank-VAT scores .631, versus .590/.589 for RRK-style/ResRank-style (both p=.0001p = .0001). Even with query alignment and retrieval similarity, ResRank-style reaches .615. Gains on audio-centric datasets support retaining complementary modality-specific evidence.

Table R6. RRK-style and ResRank-style adaptations versus FlashRank (nDCG@10).

MethodMV2MV-RAWMSR-VTTDiDeMoVATEXANetACapsVGGS-aVGGS-vAvg.
First-stage.706.356.691.585.793.605.380.318.969.600
RRK-style.694.372.645.569.794.612.361.299.966.590
ResRank-style.690.384.637.567.794.613.353.297.967.589
ResRank-style + retrieval similarity (aligned).704.413.680.598.818.650.384.312.973.615
FlashRank-omni (unaligned).703.372.682.588.811.633.380.327.973.608
VAT (without retrieval similarity).685.352.650.582.800.615.464.431.968.616
FlashRank-VAT.685.378.676.586.809.629.492.450.972.631

2. We thank the reviewer for this suggestion. We trained an independent pointwise FlashRank-VAT using the same cached representations, 7B backbone, top-100 pools, and ranking supervision, and will include the comparison in the revised manuscript (Table R7).

Findings (Table R7). Pointwise VAT scores .635 versus .631 for listwise. Listwise uses 9 rather than 100 forward passes and ∼10×\sim 10\times lower latency (.356 vs ~3.7 s/query), highlighting the efficiency of joint candidate reranking.

Table R7. Pointwise versus listwise FlashRank-VAT.

MethodAvg. nDCG@10Forward passesLatency (s/query)
Pointwise FlashRank-VAT.635100~3.7
Listwise FlashRank-VAT.6319.356

Thank you for the additional experiments. I updated my rating to 2. Although the new results clarify the effectiveness–efficiency trade-off, but they also reinforce my concern about attributing improvements to the proposed listwise architecture.

The independently trained pointwise model achieves 0.635 versus 0.631 for listwise VAT. Table R5 shows that the entire net gain from adding audio and text representations arises from the two audio-centric benchmarks. The other seven datasets show slight aggregate decline.

Even with the RRK/ResRank adaptations, it remains unclear whether they receive the same separate modality-specific representations and matched retrieval-score fusion.

We thank the reviewer for the follow-up and for reconsidering our rating. We would like to clarify that our main contribution is achieving competitive effectiveness with substantially lower latency through listwise reranking, rather than outperforming pointwise ranking in effectiveness.

Regarding RRK/ResRank, our adaptations follow their respective representation and scoring formulations, while using the same frozen index, backbone, candidate pools, and ranking supervision.

We appreciate your time and thoughtful feedback!

Reviewer_RVWJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlashRank, a listwise video reranker that directly uses cached video, audio, and text representations from a fixed multimodal retrieval index, avoiding query-time re-encoding of raw videos. It projects modality-specific representations into an LLM, contextualizes them in a single causal forward pass, and combines their scores with retrieval similarities without autoregressively generating a ranking. Training combines representation alignment with text- and video-domain ranking supervision. The query encoder can also be adapted while keeping the video index unchanged. Experiments on nine retrieval tasks demonstrate a favorable effectiveness–latency trade-off and examine representation choices, training supervision, query adaptation, and inference windows.

Critical strengths
  1. On the same top-100 candidate pools, FlashRank-VAT achieves an average nDCG@10 of 0.631, close to RankVideo's 0.627 and Qwen3VL-R8B's 0.637, with a reranking latency of only 0.356 seconds, approximately 74 times faster than RankVideo (Table 1). Appendix A.1 specifies the shared hardware, timing scope, and inclusion of frame encoding costs for the video baselines. Reusing the index and avoiding query-time video re-encoding constitute a practically meaningful contribution.

  2. Table 3 shows that, with representation alignment enabled, adding text ranking supervision improves average nDCG@10 from 0.615 to 0.631. Table 5 evaluates the approach with another retrieval index and several LLM backbones. Table 12 further shows that VAT retains an average nDCG@10 of 0.623 with a single 100-candidate window while reducing latency to 0.081 seconds. These results support the method's utility under different computational budgets.

Critical weaknesses
  1. Section 5.2 describes Concat and VAT as differing only in whether representations are combined or kept separate. However, Eq. (4) includes an additional retrieval-similarity term for VAT but not for Concat. In Table 2, with the adapted query encoder, removing this term reduces VAT's average nDCG@10 from 0.631 to 0.616, indicating a substantial effect. The latter remains above Concat's 0.607, so the results still support separate representations, but the full gap in Table 1 cannot be attributed to preserving modality identity. This limits the evidence for a central design contribution.

  2. FlashRank-VAT uses video, audio, and text information, whereas the video baselines receive 16 frames according to Appendix A.3. VAT has clear advantages on audio tasks, but scores 0.586 and 0.629 on DiDeMo and ActivityNet, compared with RankVideo's 0.756 and 0.778 (Table 1). The authors acknowledge this trade-off in Section 5.1. Nevertheless, the comparison does not clearly separate the benefits of additional modality information from those of the reranking mechanism. The substantial performance gaps on visual and temporal tasks limit the practical scope of these results. This is one reason I lean toward weak rather than clear acceptance.

Questions for authors
  1. Please clarify the actual differences between Concat and VAT in their retrieval-similarity terms in Eq. (4) and query-adaptation settings. Under otherwise identical training settings, how much of VAT's advantage remains when retrieval-similarity terms are disabled for both variants, or the same retrieval score is added to both? This would directly affect my assessment of the contribution of modality separation.

  2. Could you provide a comparison on the same candidate pools with input evidence matched as closely as possible, for example, a visual-only FlashRank variant against the current frame-based baselines? This would help distinguish gains from multimodal information from those of the reranking mechanism. I am asking for a limited ablation within the existing setup, rather than additional datasets or a new benchmark suite.

3 comments

We thank the reviewer for suggesting a visual-only ablation. FlashRank-Visual uses cached visual embeddings on the same top-100 candidate pools as the 16-frame baselines (Table R5).

Multimodal evidence versus reranking. Adding audio/text to cached visual embeddings improves FlashRank from .588 to .631 (+4.3 points), especially on AudioCaps (.301→.492)(.301 \rightarrow .492) and VGGSound-a (.240→.450)(.240 \rightarrow .450). This identifies complementary modalities as a key source of effectiveness, not superiority of the visual-only reranking mechanism.

Table R5. Visual-only FlashRank versus frame-based rerankers (nDCG@10).

MethodMV2MV-RAWMSR-VTTDiDeMoVATEXANetACapsVGGS-aVGGS-vAvg.
RankVideo (16 frames).655.381.700.756.864.778.300.217.996.627
Qwen3VL-R8B (16 frames).651.401.705.765.867.805.312.226.998.637
FlashRank-Visual.640.400.675.612.806.638.301.240.979.588
FlashRank-VAT.685.378.676.586.809.629.492.450.972.631

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results provided in response to your comments.

We would greatly appreciate it if you could take a look at our responses and let us know whether they address your concerns. We would also be happy to provide any further clarification if needed.

Thank you for the visual-only experiments in Table R5, which address my second question. However, my concern about the Concat–VAT comparison remains: differences in retrieval-score fusion and potentially query adaptation prevent attributing the full performance gap to modality separation. I maintain my original rating of 3 (weak acceptance).

Reviewer_bfsrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FlashRank, a listwise video reranker that operates directly over compressed representations already cached in a multimodal retrieval index, avoiding query-time re-encoding of raw videos. Evaluation spans nine text-to-video retrieval tasks (events, visual, audio, temporal). Ablations and generalization across two retrieval indices and four LLM backbones are reported.

Critical strengths

FlashRank is, to my knowledge, the first listwise video reranker that runs entirely over representations already stored in a fixed multimodal retrieval index, with no query-time raw-video re-encoding and no index rebuild.

The VAT/concat/omni comparison isolates the effect of preserving modality identity on identical underlying embeddings.

Critical weaknesses

Effectiveness is at parity, not superiority, and the headline comparison rests on differences likely within noise. On the main top-100 pools, average nDCG@10 of FLASHRANK_VAT (0.631) is slightly below RankVideo-video (0.637).

The gains depend on the index storing modality-specific embeddings: when only a joint embedding is available (FLASHRANK_omni, 0.608), the advantage over the first stage (0.600) largely disappears, so the benefit is tied to a particular index design (OmniEmbed-MV).

Questions for authors

Table 1's First-stage row reports DiDeMo .785, ACaps .318, VGGS-a .969, VGGS-v .600, while Tables 2 and 12 report DiDeMo .585, ACaps .380, VGGS-a .318, VGGS-v .969 with an average of .600 that is internally consistent — which values are correct? Relatedly, in Table 1 the video-mode Qwen3VL-R8B row is numerically identical to RankVideo's across all nine datasets, and both RankVideo and Qwen3VL-R8B are listed at exactly 26.3s in both caption- and video-mode (the main text also cites 26.8s and 75× for Qwen3-VL-R8B, inconsistent with the table's 26.3s).

The central claim is effectiveness parity at ~74× lower latency, but no significance testing or run-to-run variance is reported (differences such as 0.631 vs 0.637 are small). Can you provide standard deviations/significance tests for the headline comparisons?

All evaluations use top-100 pools over corpora that are small on 7/9 tasks. Do the findings hold for larger candidate pools (e.g., top-1000) and larger-scale indexes, where the latency argument is strongest?

2 comments

1. Effectiveness, efficiency, and index dependence

We thank the reviewer for highlighting effectiveness–efficiency and index dependence. To directly test effectiveness parity, we added Two One-Sided Tests (TOST) on paired query results (±1\pm 1 nDCG@10 point). FlashRank-VAT achieves equivalent nine-dataset macro-average effectiveness to RankVideo at ∼74×\sim 74\times lower latency (Table R1). VAT benefits from modality-specific index embeddings; with joint-only representations, FlashRank-omni still improves over First-stage (.608 vs .600), albeit less.

Table R1. Effectiveness and reranking latency (nine-dataset macro-average).

MethodAvg. nDCG@10Latency (s/query)
RankVideo.62726.3
Qwen3VL-R8B.63726.8
FlashRank-VAT.631.356

VAT vs. RankVideo: Δ\Delta = +0.34 nDCG@10 point, 90% CI [−0.29, +0.98], TOST p=.045p = .045 (±1\pm 1-point equivalence margin).


2. Consistency of reported results

We thank the reviewer for checking the reported results. We reverified the originally submitted PDF and found the tables internally consistent.

First-stage scores agree across Tables 1, 2, and 12: DiDeMo .585, AudioCaps .380, VGGSound-a .318, VGGSound-v .969 (Avg. .600).

RankVideo-video and Qwen3VL-R8B-video have distinct average scores (.627/.637) and latencies (26.3/26.8 s/query); Qwen3VL-R8B-caption takes 2.63 s/query. The ∼74×\sim 74\times/∼75×\sim 75\times speedups are consistent with VAT's .356 s/query.


3. Statistical significance and variance

We added three-seed training runs and paired statistical tests on the same nine-dataset top-100 candidate pools.

Run-to-run stability: Ranking-stage seeds 42/1/2 yield .631/.629/.628 nDCG@10 (0.6294±0.00130.6294 \pm 0.0013, mean ± SD), keeping the alignment checkpoint, training data, and hyperparameters fixed. Both new seeds are equivalent to the reference run within the ±1\pm 1-point macro-average TOST margin.

Statistical testing protocol: We use paired query-level results: 90% CIs from 10,000 stratified bootstrap resamples (the conventional CI level for TOST at α=.05\alpha = .05), difference p-values from 10,000 two-sided paired permutations, and TOST p-values to test equivalence within ±1\pm 1 nDCG@10 point.

Findings (Table R2): VAT significantly improves over First-stage, Concat, and Omni (all p=.0001p = .0001). Claims256 gives additional effectiveness gains at higher computational cost.

Table R2. Statistical significance and equivalence (nine-dataset macro-average).

ComparisonΔ\Delta nDCG@10 (points)90% CIDifference pTOST p
VAT vs. First-stage+3.05[+2.64, +3.48].00011.000
VAT vs. Concat+2.42[+2.02, +2.82].00011.000
VAT vs. Omni+2.31[+1.88, +2.75].00011.000
VAT + Claims256 vs. First-stage+4.86[+4.42, +5.31].00011.000
VAT vs. RankVideo+0.34[−0.29, +0.98].416.045
VAT + Claims256 vs. RankVideo+2.15[+1.53, +2.76].0001.999

Δ\Delta = first method minus second (×100). Difference p<.05p < .05 indicates a significant difference; TOST p<.05p < .05 establishes equivalence within ±1\pm 1 point. The smallest reported permutation p-value is .0001.


4. Scalability to larger candidate pools

We thank the reviewer for asking about scalability. Because FlashRank reranks cached embeddings, online cost mainly follows candidate depth and window count rather than total index size. We additionally evaluated ActivityNet top-1000 candidates (Table R3).

Findings (Table R3): With 20/10 windows, top-1000 versus top-100 yields .627 versus .629 nDCG@10. At top-1000, 100/50 windows take 1.50 rather than 4.20 s/query (nDCG@10 .622).

Table R3. ActivityNet top-100 versus top-1000: effectiveness and reranking latency.

Candidate depthWindow / StrideForward passesnDCG@10Measured latency (s/query)
Top-100 (original)20 / 1090.6290.386
Top-100020 / 10990.6274.20
Top-1000100 / 50190.6221.50

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results provided in response to your comments.

We would greatly appreciate it if you could take a look at our responses and let us know whether they address your concerns. We would also be happy to provide any further clarification if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.