- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SelfRank, a highly efficient reranking method for low-latency visual document retrieval. Instead of relying on gradient-based training, hyperparameter tuning, or relevance labels, SelfRank uses a closed-form rule to optimally combine retriever scores and compressed reranker scores by maximizing correlation with relevance. Furthermore, it restores missing relevance information in the compressed model by fitting a shallow linear readout to the model's own uncompressed scores using a centered ridge regression. Evaluated on ViDoRe datasets, it achieves near-full-model accuracy while running up to 48x faster.
Critical strengths
-
An interesting and efficient method. SelfRank combines closed-form readout fitting with retriever–reranker score fusion. It helps recover ranking quality after compression without gradient-based training. By keeping the backbone frozen, the adaptation is highly lightweight.
-
Solid experiments. The experiments cover 12 datasets from ViDoRe 2 and ViDoRe 3, two retrievers, and two multimodal backbones. This breadth, together with the component ablations, provides strong evidence of the method’s benefits .
Critical weaknesses
-
Unclear presentation. The Method section does not follow the actual workflow: self-supervised target construction, centered ridge regression, and finally score fusion. This organization increases the reader’s burden. Some symbols, such as , are also used without explaining their meaning upon first introduction. The authors should reorganize the Method section and clarify the notation.
-
Unclear reranker setup and reproducibility. The experiments use general MLLMs as rerankers, but the paper does not clearly state whether they have undergone relevance-specific fine-tuning. Off-the-shelf MLLMs typically perform poorly in zero-shot document reranking without relevance-specific fine-tuning. It is unclear how the authors elicited such strong baseline performance. Please disclose the exact scoring prompts and inference configurations to ensure reproducibility.
Questions for authors
-
Configuration. What exact instruction prompts were used, and did the Qwen and GLM backbones undergo any relevance-specific fine-tuning? Furthermore, regarding the latency results in Figure 3, what exactly do "Retrieval", "Page load", and "Compute" entail? Why are certain components missing for some models? Please provide the detailed inference configurations ( such as, bsz, input lengths, precision, etc.) to ensure these latency comparisons are fully transparent and reproducible.
-
Backbone comparisons and efficiency. Can the authors provide matched accuracy and efficiency comparisons between the original Qwen2.5-VL and GLM-4.1V rerankers and their SelfRank counterparts? Additionally, why is the 7B Qwen-based SelfRank faster than Qwen3-Reranker-0.6B in the experiments?
-
Smaller-model evaluation. Given the stringent latency requirements of industrial reranking, how does SelfRank perform on smaller models, such as Qwen3.5-0.8B?
3 comments
Thank you to the authors for their response. Most of my concerns have been addressed, and I will raise my score.
Thank you for checking our revision, and for raising your score!
Thank you for the detailed review and the concrete requests. All of them are addressed in the revision.
§3 now follows the pipeline:
Revision: §3 is reordered. Fig. 1 (its section badges) and Alg. 1 follow the same order, every symbol is defined in §3.1 at first use, and App. C adds a notation table (Table 10).
No relevance fine-tuning on either backbone, and no gradient step anywhere.
Weights:
Prompt. Applied through each model's chat template, with the image first:
Score. logit(Yes) − logit(No) at the first answer position, with no decoding. For GLM we read it at
<|assistant|>\n, before<think>.Why zero-shot is strong. The model only reorders 20 candidates from a strong retriever. ViDoRe queries are also close to VQA, which these models are trained on. As evidence, the zero-shot Qwen-7B CE scores 71.98 on ViDoRe 2, on par with the relevance-trained Qwen3-VL-Reranker-8B (71.83). On ViDoRe 3, the 8B is ahead (60.29 vs. 55.89).
Revision: new App. D, "Reproducibility Details" (prompt, weights, scoring, Table 11). §4 states that neither backbone is fine-tuned.
Definitions:
Why segments are missing. The retrievers do no reranking, and GQR uses embeddings already in memory, so their page-load segments are empty. For the full CE, loading was wrongly merged into compute. The revised figure separates them, using the re-timed GLM CE: 807 ms load and 2974 ms compute.
Revision: Fig. 3 is redrawn and its caption names the CE. §4.2 states what each component covers.
Revision: Table 13 (App. F), which also has the unfused rows.
Why the 7B is faster than the 0.6B. Latency follows the tokens processed:
Revision: App. H.
Revision: new App. I (Table 17).