acceptodds
Under review as a conference paper at ICLR 2027

SelfRank: Closed-Form Reranking for Low-Latency Multimodal Retrieval

Abstract

Reranking with a multimodal language model orders visual document retrieval accurately but slowly. Some existing methods require relevance labels to recover accuracy after compression and ignore retriever scores. We propose SelfRank, a reranker built on a closed form rule that quantifies and restores the missing relevance information. We derive the optimal operating point under a correlation objective and establish the exact condition under which reranker scores alone are insufficient to achieve it. SelfRank further restores the reranker with a single vector on its shallow state, fitted to the model's own uncompressed score by one centered ridge solve. Across 12 datasets from ViDoRe 2 and ViDoRe 3 with two retrievers and two backbones, SelfRank comes within 1.2 pp of a full cross encoder in NDCG@5 while running up to 48x faster, pushing the Pareto frontier of accuracy against latency in visual document reranking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal document retrieval, reranking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EmpsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SelfRank, a highly efficient reranking method for low-latency visual document retrieval. Instead of relying on gradient-based training, hyperparameter tuning, or relevance labels, SelfRank uses a closed-form rule to optimally combine retriever scores and compressed reranker scores by maximizing correlation with relevance. Furthermore, it restores missing relevance information in the compressed model by fitting a shallow linear readout to the model's own uncompressed scores using a centered ridge regression. Evaluated on ViDoRe datasets, it achieves near-full-model accuracy while running up to 48x faster.

Critical strengths
  1. An interesting and efficient method. SelfRank combines closed-form readout fitting with retriever–reranker score fusion. It helps recover ranking quality after compression without gradient-based training. By keeping the backbone frozen, the adaptation is highly lightweight.

  2. Solid experiments. The experiments cover 12 datasets from ViDoRe 2 and ViDoRe 3, two retrievers, and two multimodal backbones. This breadth, together with the component ablations, provides strong evidence of the method’s benefits .

Critical weaknesses
  1. Unclear presentation. The Method section does not follow the actual workflow: self-supervised target construction, centered ridge regression, and finally score fusion. This organization increases the reader’s burden. Some symbols, such as pp, are also used without explaining their meaning upon first introduction. The authors should reorganize the Method section and clarify the notation.

  2. Unclear reranker setup and reproducibility. The experiments use general MLLMs as rerankers, but the paper does not clearly state whether they have undergone relevance-specific fine-tuning. Off-the-shelf MLLMs typically perform poorly in zero-shot document reranking without relevance-specific fine-tuning. It is unclear how the authors elicited such strong baseline performance. Please disclose the exact scoring prompts and inference configurations to ensure reproducibility.

Questions for authors
  1. Configuration. What exact instruction prompts were used, and did the Qwen and GLM backbones undergo any relevance-specific fine-tuning? Furthermore, regarding the latency results in Figure 3, what exactly do "Retrieval", "Page load", and "Compute" entail? Why are certain components missing for some models? Please provide the detailed inference configurations ( such as, bsz, input lengths, precision, etc.) to ensure these latency comparisons are fully transparent and reproducible.

  2. Backbone comparisons and efficiency. Can the authors provide matched accuracy and efficiency comparisons between the original Qwen2.5-VL and GLM-4.1V rerankers and their SelfRank counterparts? Additionally, why is the 7B Qwen-based SelfRank faster than Qwen3-Reranker-0.6B in the experiments?

  3. Smaller-model evaluation. Given the stringent latency requirements of industrial reranking, how does SelfRank perform on smaller models, such as Qwen3.5-0.8B?

3 comments

Thank you for the detailed review and the concrete requests. All of them are addressed in the revision.

W1: The Method section does not follow the actual workflow [...]. Some symbols, such as (p), are also used without explaining their meaning upon first introduction.

§3 now follows the pipeline:

  1. Setting and serving: defines qq, a candidate page p∈Cqp \in \mathcal C_q, KK, bq\mathbf b_q, sq\mathbf s_q, ℓ\ell and γ\gamma.
  2. Self-supervised target.
  3. Centered ridge.
  4. Fusion.

Revision: §3 is reordered. Fig. 1 (its section badges) and Alg. 1 follow the same order, every symbol is defined in §3.1 at first use, and App. C adds a notation table (Table 10).

W2/Q1: [...] whether they have undergone relevance-specific fine-tuning [...]. What exact instruction prompts were used [...]? Please provide the detailed inference configurations.

No relevance fine-tuning on either backbone, and no gradient step anywhere.

Weights:

  • Qwen2.5-VL-7B-Instruct;
  • GLM-4.1V-9B-Thinking (original release).

Prompt. Applied through each model's chat template, with the image first:

Does this document page contain information that answers the query?
Query: {q}
Answer only Yes or No.

Score. logit(Yes) − logit(No) at the first answer position, with no decoding. For GLM we read it at <|assistant|>\n, before <think>.

Full CE (teacher)SelfRank
Page input≤768 visual tokens256 cached visual positions + text
Query inputfull promptquery suffix only (~36 tokens)
Layers28 / 4021
Precisionbf16bf16, int8 KV store
Batch20 candidates20 candidates
Hardware1x A100 40GB1x A100 40GB

Why zero-shot is strong. The model only reorders 20 candidates from a strong retriever. ViDoRe queries are also close to VQA, which these models are trained on. As evidence, the zero-shot Qwen-7B CE scores 71.98 on ViDoRe 2, on par with the relevance-trained Qwen3-VL-Reranker-8B (71.83). On ViDoRe 3, the 8B is ahead (60.29 vs. 55.89).

Revision: new App. D, "Reproducibility Details" (prompt, weights, scoring, Table 11). §4 states that neither backbone is fine-tuned.

Q1 (cont.): In Figure 3, what exactly do "Retrieval", "Page load", and "Compute" entail? Why are certain components missing?

Definitions:

  • Retrieval: first-stage search for the top 20 (shared by all methods).
  • Page load: reading what a method keeps for each candidate. For SelfRank this is the stored KV prefix and its dequantization; for the image rerankers it is the image files.
  • Compute: everything after loading, including image preprocessing and the forward passes.

Why segments are missing. The retrievers do no reranking, and GQR uses embeddings already in memory, so their page-load segments are empty. For the full CE, loading was wrongly merged into compute. The revised figure separates them, using the re-timed GLM CE: 807 ms load and 2974 ms compute.

Revision: Fig. 3 is redrawn and its caption names the CE. §4.2 states what each component covers.

Q2: matched accuracy and efficiency comparisons between the original Qwen2.5-VL and GLM-4.1V rerankers and their SelfRank counterparts? [...] why is the 7B Qwen-based SelfRank faster than Qwen3-Reranker-0.6B?

ms/queryV2 ColNomicV2 ColPaliV3 ColNomicV3 ColPali
Qwen-7B full CE + fusion348472.1267.3657.8751.27
SelfRank (Qwen-7B)12869.1265.2256.2248.90
GLM-9B full CE + fusion378172.7068.5458.3351.05
SelfRank (GLM-9B)9171.2266.1255.7648.75

Revision: Table 13 (App. F), which also has the unfused rows.

Why the 7B is faster than the 0.6B. Latency follows the tokens processed:

  • The 0.6B reads about 740 OCR tokens per page, about 14.8k tokens per query (~17.8 TFLOP).
  • SelfRank runs only the 36-token query against each cached page, about 720 tokens per query (~7.5 TFLOP).

Revision: App. H.

Q3: how does SelfRank perform on smaller models, such as Qwen3.5-0.8B?

ms/queryV2 ColNomicV2 ColPaliV3 ColNomicV3 ColPali
Qwen3.5-0.8B full CE200167.8763.6955.3746.77
SelfRank (Qwen3.5-0.8B)6367.7863.3355.2146.58
SelfRank (GLM-4.1V-9B)9171.2266.1255.7648.75

Revision: new App. I (Table 17).

Thank you to the authors for their response. Most of my concerns have been addressed, and I will raise my score.

Thank you for checking our revision, and for raising your score!

Reviewer_FohTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SelfRank - a method for low latency re-ranking of visually rich (multimodal documents).

The main ideas behind the method include: (1) Storing the KV cache that corresponds to the task+page tokens offline, appending only the query tokens at inference time (2) Predicting the full LLM score from an early-exit hidden state through ridge regression (3) Fusing the distilled score with a first-stage retriever score through CombSUM, with the fusion weights smartly selected.

SelfRank is then evaluated against ViDoRe 2 and ViDoRe 3, yielding consistent gains over baselines, closing the gap to a full cross encoder performance. The authors further performed multiple ablation studies on the early-exit layer, number of candidates per query and the different components of the method.

Critical strengths
  1. The method seems to have strong performance on standard benchmarks, closing the gap to full cross-encoder pass, though much faster.
  2. The derivation of the fusion weight that maximizes correlation is nice.
Critical weaknesses
  1. The paper lacks a lot of important details. I have critical concerns about the feasibility of the offline caching, which was discussed very little in the paper and seems to be a major reason for the speedup.
  2. The novelty of the proposed components is marginal. Specifically, similar ideas in query-independent precomputation in and early in transformers exit were thoroughly investigated. Using "yes"/"no" logit diff is standard practice when using LLMs for pointwise reranking. It is also well known that the intermediate ("shallow") representations of LLMs may be more useful than the final representation.
Questions for authors

I have major questions and concerns that I would appreciate the authors addressing:

  1. There was a very little discussion regarding the offline token caching which I believe is a major reason for the speedup. Specifically:
  • I believe that storing the KV cache offline for the whole corpus as discussed in the paper is infesiable. Even with the early exit and quantization to int8, storing all of keys and values amounts to storing 21⋅2=4221\cdot2=42 full sequences of tokens. Storing even a single token sequence (as in multivector methods e,g ColPali) is considered very expensive and impractical, requiring tokens to be pruned. Am I missing something? What is the storage required per page (and how is it compared to the storage required by ColPali etc.)? Is the intended storage RAM or disk? What's the time associated with loading the page?
  • How much of the forward-pass speedup is attributed to the caching vs. the early exit?
  • Regarding the same matter, do only one third of the visual tokens are considered? I am slightly confused by the statement in line 300
  1. In proposition 1, the objective to be maximized is ⟨f,y⟩⟨f,f⟩\frac{\langle f,y \rangle}{\sqrt{\langle f,f \rangle}}, and not ⟨f,y⟩\langle f,y \rangle which has a trivial endpoint maximizer. The authors should explicitly state this in the main text and justify this as well. In addition, were cbc_b, csc_s and ρ\rho actually estimated using the training data? I am not sure that the selection of the fusion coefficient w in the experiments was discussed at all. Is this a new result? have you checked for similar results in the classic IR literature (as this is quite a general statement about CombSUM)?

  2. In the experiments, which model was used as the reference "full cross encoder"? I didn't find it in the paper although SelfRank was reported with both Qwen2.5-VL-7B and GLM-4.1V-9B. Also, comparing SelfRank with a standalone pointwise-LLM isn't completely fair because SelfRank fuses its score with the retriever score. I believe that a fair comparison would be to compare the full SelfRank method with a "Full cross encoder + Retriever score sum fusion" (at the operating point w∗w^*), as well as SelfRank without fusion (i.e only the estimated shallow pass score) vs the Full cross encoder.

4 comments

Thank you. The storage cost was under-reported in the submission; the revision states it in §3.1 and §4.2 and gives the full numbers in App. G.

Q1: storing the KV cache offline for the whole corpus [...] is infeasible [...]. What is the storage required per page (and how is it compared to [...] ColPali etc.)? Is the intended storage RAM or disk? What's the time associated with loading the page?

At 4 bits the store is 1.7–3.0 MB per page. It lives on disk and is read only for the 20 candidates. This is 6–11x a ColPali index, and we state it as the method's main cost.

MB per pageint8 (submitted)4-bit (new)1M pages, 4-bit
ColPali index0.26—0.26 TB
Page image (JPEG)0.5–0.7—0.5–0.7 TB
SelfRank, GLM-9B3.051.661.7 TB
SelfRank, Qwen-7B6.322.953.0 TB

Why smaller than "2ℓ2\ell sequences":

  • 256 of 768 visual positions are kept;
  • grouped-query attention stores only 4 (Qwen) or 2 (GLM) KV heads;
  • keys and values are int8.

RAM vs. disk. A ColPali index is scanned for every query, so it must sit in RAM. Our store is read only for the 20 retrieved candidates, which makes it a disk workload, like fetching 20 page images.

Load time (20 pages):

Warm (OS cache)Cold (network virtual disk)
GLM7 ms of 91 ms total288 ms total
Qwen24 ms of 128 ms total403 ms total

Even fully cold, SelfRank is 4.7–6.6x faster than Qwen3-VL-Reranker-2B and 7.7–10.7x faster than the 8B.

4-bit store (KIVI-style: keys per channel, values per token):

  • Qwen: unchanged in all 4 cells (within ±0.4 pp).
  • GLM: 3 of 4 cells unchanged; one loses 0.96 pp.

Revision: §3.1 and §4.2 state that the store lives on disk and is read only for the candidates of a request. New App. G gives per-page and per-collection storage (Table 14), warm and cold load times (Table 15) and the low-bit stores (Table 16).

Q2: How much of the forward-pass speedup is attributed to the caching vs. the early exit?

Mostly caching. Early exit adds 1.3–1.9x on top of it and cuts storage by 25–47%.

ms/query, 20 candidatesQwen-7BGLM-9B
Full CE34843781
Early exit only3265 (1.07x)3110 (1.2x)
Cache only162 (21x)173 (22x)
Both (SelfRank)128 (27x)91 (41x)

Revision: App. G (Table 15).

Q3: do only one third of the visual tokens are considered? [...] line 300

Yes. The cache keeps 256 of 768 visual positions (farthest-point sampling) plus all text. The teacher CE sees the full page.

Revision: the sentence in §4 ("Datasets and models") is rewritten to say exactly this.

Q4: the objective [...] is corr(f,y)\mathrm{corr}(\mathbf f,\mathbf y), and not ⟨f,y⟩\langle\mathbf f,\mathbf y\rangle [...]. Were cbc_b, csc_s, ρ\rho estimated using the training data? [...] Is this a new result? [...] classic IR literature?

  • Objective. Yes, it is the correlation, as stated in Prop. 1. We now state the choice in the main text and justify it. ⟨f,y⟩\langle\mathbf f,\mathbf y\rangle is linear in ww, so its maximum is at an endpoint. Rankings are scale-invariant, so the objective must be scale-invariant too.
  • Estimation. ViDoRe has no training split. The three correlations are estimated on the next corpus of the same benchmark, using its labels, so no corpus sets its own weight. The baselines tune their weights on the same corpus. A label-free variant (App. B.5, Table 9) reaches 59.9 vs. 60.1.
  • Novelty. The form is not new. It is the classical optimal linear combination of two predictors (Bates & Granger, 1969). Linear fusion was also studied in IR (Vogt & Cottrell, 1999). We now say so and cite both. What we add:
    • a deployment rule, with no sweep: predicted vs. measured optimal ww gives r = 0.898 over 72 subsets;
    • a testable condition, cb>ρcsc_b > \rho c_s. It holds even for the uncompressed CE, which is optimal at w=1w = 1 in 0 of 24 subsets (median ww = 0.80).

Revision: §3.4 (now Prop. 2) adds three things:

  • the choice of the correlation objective, stated and justified just before the proposition;
  • a paragraph "Estimating the weight";
  • a paragraph "Relation to classical results", citing Bates & Granger (1969) and Vogt & Cottrell (1999).

Q5: which model was used as the reference "full cross encoder"? [...] a fair comparison would be [...] SelfRank with "Full cross encoder + Retriever score sum fusion" [...], as well as SelfRank without fusion vs the Full cross encoder.

The reference was Qwen2.5-VL-7B at full depth, already fused with the retriever. The caption was wrong and has been corrected. Your proposed comparison, for each backbone against its own CE:

CE aloneCE + fusionReadout aloneSelfRank
Qwen, V2 ColNomic / ColPali71.98 / 67.0072.12 / 67.3667.73 / 64.7369.12 / 65.22
Qwen, V3 ColNomic / ColPali55.89 / 50.7557.87 / 51.2751.72 / 47.6556.22 / 48.90
GLM, V2 ColNomic / ColPali72.29 / 67.9372.70 / 68.5469.64 / 65.1471.22 / 66.12
GLM, V3 ColNomic / ColPali55.80 / 50.4658.33 / 51.0551.99 / 47.9055.76 / 48.75

Gaps:

  • Unfused vs. unfused: 2.3–4.3 pp.
  • Fused vs. fused: 1.5–3.0 pp. Fusion helps the readout more than the CE (+0.5 to +4.5 vs. +0.1 to +2.5 pp), so part of what compression loses is supplied by the retriever score. This is what Prop. 1 predicts: the weaker the reranker score, the more the retriever's independent information is worth. Fusion is therefore part of the method, not an add-on.

SelfRank remains above all external baselines in Tables 1–2 and is 27–41x faster than its own CE.

Revision: Tables 1 and 2 now list each backbone's own fused CE, with corrected captions, and §4.1 compares against it. Table 13 (App. F) has all four rows per backbone.

Novelty: query-independent precomputation, early exit, yes/no logit diff and shallow representations are known.

Agreed; the paper claims none of these (§2). We also do not claim that shallow states are better: layer 21 is used only because it is cheap. Our contributions are:

  1. Label-free, closed-form recovery. One ridge solve on 154 unlabeled queries, fitted to the model's own full-depth score. It beats reading layer 21 directly (the logit lens) by 1.2–2.9 pp on ViDoRe 2.
  2. The fusion condition above.
  3. Composability. On Qwen3-VL-Reranker-8B, it reaches ViDoRe 2 parity at 12.5x lower latency.

I thank the authors for their detailed and careful response, conducting all of the experiments I have asked for.

However, the new results confirm my main concern. Most of the efficiency gain comes from precomputing and storing the KV cache for the whole corpus and not from the main contributions of the paper. Even after visual token pruning, GQA and quantization the storage is orders of magnitudes higher compared to methods like ColPali which by themselves usually utilize pruning.

Every latency comparison in the main paper assumes warm storage. At these sizes, keeping a production corpus warm is not realistic. The authors' own cold numbers show latency rising about 3.1x.

Given the above, and after an overall consideration, my verdict remains negative.

Thank you for the follow-up. All five points from your initial review are addressed in the revised PDF:

  • Q1 (Storage, RAM vs. disk, load time): Added disk-serving details in Sections 3.1 and 4.2, plus per-page storage, warm/cold timings, and 2-4 bit stores in Appendix G (Tables 14-16).
  • Q2 (Cache vs. early-exit speedup): Added the latency breakdown in Table 15.
  • Q3 (Visual token budget): Rewrote Section 4 to state 256 of 768 visual positions plus all text positions.
  • Q4 (Prop. 2 objective, estimation, prior IR work): Added the objective justification, weight estimation, and citations to Bates & Granger (1969) and Vogt & Cottrell (1999) in Section 3.4.
  • Q5 (Same-backbone CE comparisons): Fixed captions and added each backbone's fused/unfused CE alongside unfused readouts in Tables 1-2, Section 4.1, and Table 13.

We would like to clarify the three points in your follow-up:

  1. "Most of the efficiency gain comes from precomputing and storing the KV cache ... not from the main contributions." The compressed cache (1/3 visual tokens, 25-47% fewer layers, 3-4 bit) does not work on its own. Without our fusion, it scores 51.7-52.0 on ViDoRe 3 ColNomic (below 52.8 without reranking); without our readout, it drops up to 2.9-4.1 pp on ViDoRe 2 (Tables 3, 13). The closed-form readout and fusion recover that loss without labels or fine-tuning, and early exit cuts cache size by 25-47%.

  2. "the storage is orders of magnitudes higher compared to methods like ColPali" At 3-4 bit, GLM uses 1.29-1.66 MB/page (within 0.37 pp at 3-bit, Table 16). That is 5-6x ColPali (0.26 MB/page), not orders of magnitude. ColPali sits in RAM because every query scans the full index; our prefix sits on disk next to the JPEG page images (0.46-0.66 MB/page, Table 14) and is read only for 20 candidates (2-3.6x JPEG).

  3. "keeping a production corpus warm is not realistic ... cold numbers show latency rising about 3.1x." Even completely cold on a network disk (cache evicted before every query: 288 ms, GLM int8, Table 15), SelfRank is still 13.1x faster than Full CE and 6.6-10.7x faster than Qwen3-VL-Reranker-2B/8B (all timed warm). The Pareto gain does not depend on a warm cache.

Reviewer_Qz7SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces the SelfRank, fits a shallow linear readout to a multimodal reranker’s full-depth scores using query-centered ridge regression, then combines its predictions with retriever scores through a correlation-derived fusion weight. It builds on visual-prefix caching, early exit, and cache compression. Experiments cover twelve ViDoRe corpora, two retrievers, and two backbones, reporting substantial latency reductions with relatively small ranking losses.

Critical strengths
  1. The single-vector readout offers a simple, inexpensive way to recover quality after compression without updating the backbone or requiring relevance labels for fitting. The controlled ablation shows a 4.1-point NDCG@5 improvement over the unfitted logit lens (Table 3), supporting the value of teacher-score calibration.
  2. The readouts transfer from ColNomic candidates on ViDoRe 2 to another retriever and ViDoRe 3. This provides useful evidence that the fitted readout generalizes beyond its calibration distribution.
Critical weaknesses
  1. The optimality argument does not directly apply to the deployed fusion. Section 3.1 estimates correlations using discount-weighted standardization, while the final fusion retains ordinary z-scores from Eq. 1. The weighted proof in Appendix A.1 requires consistent normalization. Switching between these normalizations introduces score-dependent variance factors that Eq. 4 does not account for. This leaves the central closed-form operating-point claim insufficiently justified.
  2. The claimed Pareto improvement is not established against the closest alternatives. The evaluation omits miniReranker and Qwen3-VL-Reranker, despite citing both. Moreover, the headline 1.2-point gap and 48× speedup compare the GLM-9B readout with full Qwen-7B, rather than its corresponding full GLM teacher. These comparisons do not establish the incremental benefit over existing efficient multimodal rerankers or recovery relative to the same backbone.
Questions for authors
  1. How is the normalization mismatch between weight estimation and deployed fusion resolved? Does consistent normalization, or the corresponding variance correction, change the reported results?
  2. How does SelfRank compare with miniReranker and Qwen3-VL-Reranker under matched candidate pools, image budgets, timing boundaries, and retriever-score fusion?
3 comments

Thank you. Both points are valid, and both are fixed.

W1/Q1: Section 3.1 estimates correlations using discount-weighted standardization, while the final fusion retains ordinary z-scores from Eq. 1. [...] How is the normalization mismatch [...] resolved? Does consistent normalization, or the corresponding variance correction, change the reported results?

The mismatch was real. Fixing it changes no cell by more than 0.3 pp.

Per query, the two standardizations differ by:

  • a shift, which does not affect the ranking;
  • a scale ratio, which does. Because of it, the deployed fusion effectively used a query-dependent weight instead of w⋆w^\star.

We now fuse with the same weighted standardization used for estimation, so the proof in App. A.1 applies unchanged.

Macro NDCG@5Per-cell change vs. submitted
Submitted60.12—
Consistent normalization (adopted)60.16−0.09 to +0.28
Variance correction instead60.19similar

Revision: Eq. 1 (§3.1) now defines the weighted standardization, used for both estimation and fusion (§3.4). All main-text tables are recomputed under it (Tables 1 and 2). The per-cell comparison, including the variance correction, is in App. E (Table 12).

W2/Q2: The evaluation omits miniReranker and Qwen3-VL-Reranker [...]. How does SelfRank compare [...] under matched candidate pools, image budgets, timing boundaries, and retriever-score fusion?

All methods share:

  • the same 20 candidates;
  • at most 768 visual tokens per page;
  • the same timing (page files → 20 scores on one A100, retrieval excluded);
  • the same fusion rule.
Rerankerms/queryV2 ColNomicV2 ColPaliV3 ColNomicV3 ColPali
Qwen3-VL-Reranker-2B189965.3761.8856.7450.44
Qwen3-VL-Reranker-8B309571.8368.1660.2953.86
miniReranker (GLM) §9168.3263.9255.4548.13
SelfRank (GLM-4.1V-9B)9171.2266.1255.7648.75
SelfRank (Qwen2.5-VL-7B)12869.1265.2256.2248.90
SelfRank (Qwen3-VL-Reranker-8B)24772.0968.0959.2452.79

vs. 2B. SelfRank is better by 3.3–5.9 pp on ViDoRe 2 and worse by 0.5–1.7 pp on ViDoRe 3. It is 15–21x faster.

vs. 8B. The 8B is more accurate than SelfRank on our zero-shot backbones. Applied to the 8B itself, SelfRank:

  • matches it on ViDoRe 2 (+0.26 and −0.08 pp);
  • is 12.5x faster.

In that setup the page is placed first and 21 of 36 layers are cached in int8 with no token pruning.

vs. miniReranker. Its weights and code are not released, so we reimplemented it on our backbones: page cache, exit at layer 21, and scoring with the model's own yes/no unembedding. SelfRank is better in 7 of 8 cells (up to +2.9 pp) and equal in the remaining one (−0.04).

§ Our reimplementation of its inference pipeline; its relevance fine-tuning is not included, as no checkpoint is released.

Revision: new App. F with Table 13 (all rows above, plus each backbone's own CE), including the setup of the 8B and of the miniReranker reimplementation. The baseline paragraph of §4 points to it.

W2 (cont.): the headline 1.2-point gap and 48× speedup compare the GLM-9B readout with full Qwen-7B, rather than its corresponding full GLM teacher.

Agreed; the headline is corrected. We ran the full GLM CE on all pools and re-timed it at its fastest setting (FlashAttention-2, batch 20).

GLM SelfRank vs. full GLM CE (both fused)V2 ColNomicV2 ColPaliV3 ColNomicV3 ColPaliSpeedup
Gap−1.48−2.42−2.57−2.3041x

New abstract: "...SelfRank comes as close as 1.5 pp to the full cross encoder of the same backbone in NDCG@5 while running up to 41x faster..."

Revision: the abstract, the end of §1, contribution 3 and the conclusion now state this same-backbone claim. Tables 1 and 2 list each backbone's own fused CE, and §4.1 compares against it. §4.2, Figs. 2–3 and Table 4 (App. B) use the re-timed CE.

Thank you for the detailed response and revisions. These updates address several points raised in my review, but they do not sufficiently change my overall assessment. I will therefore maintain my original score.

Thank you for reviewing our response and revision.

We noted that your original review raised two critical weaknesses: (1) the normalization mismatch between weight estimation and deployed fusion, and (2) missing matched comparisons against miniReranker and Qwen3-VL-Reranker along with the cross-backbone headline. We believe we have resolved both in the revised manuscript, e.g., unified weighted normalization in Eq. 1 and Table 12 with <= 0.3 pp change; matched evaluations against Qwen3-VL-Reranker-2B/8B, miniReranker, and SelfRank applied to the 8B in Table 13; and the corrected same-backbone headline throughout.

Could you please kindly share what specific remaining concern keeps your overall assessment negative? We would be glad to clarify or address it before the discussion closes.

Comments on the paper

1 comment

We thank all reviewers. We uploaded a revised PDF and all changes are in blue. Main changes:

  1. Same-backbone headline (all reviewers). Against its own full cross encoder (CE), with the CE also fused with the retriever, SelfRank is 1.5 pp lower and 41x faster. The cross-backbone "1.2 pp / 48x" claim is removed.
  2. Consistent normalization (Qz7S). Prop. 1 now applies to the deployed fusion exactly. Macro NDCG@5 moves from 60.12 to 60.16.
  3. New baselines (Qz7S): Qwen3-VL-Reranker-2B/8B and miniReranker (reimplemented), under matched settings. SelfRank applied to the 8B matches it on ViDoRe 2 at 12.5x lower latency.
  4. Storage and serving (FohT). A sentence in §4.2 and a new App. G cover:
    • size per page;
    • disk vs. RAM;
    • load time;
    • cache vs. early exit;
    • a 4-bit store that halves storage.
  5. Prop. 1 (FohT). We state the objective in the main text, explain how ww is estimated, and add its relation to classical results.
  6. Clarity (Emps). The Method section is reordered. We add the prompt and configuration, define the terms in Fig. 3, and report results on Qwen3.5-0.8B.

Because Section 3 now follows the pipeline, the fusion proposition (Prop. 1 in the submission) is numbered Proposition 2 in the revision.

All numbers below are NDCG@5 (%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.