- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that mainstream rerankers select by relevance, yielding individually relevant but collectively redundant or incomplete sets. It also identifies a key limitation of the rubric-reward approach: the reward is a single scalar shared by every document, making contributors indistinguishable from free riders. To address this, the paper extends RubricReranker and proposes AdaTutoRank, a setwise reranker trained with Adaptive Tutoring Optimization (ATO) under a three-level rubric hierarchy of nine dimensions that covers document-level, set-level, and global set properties. These rubrics are carried through the entire training pipeline, serving as silver labels for cold-start SFT, as rewards for reinforcement learning, and as hints for distillation, which makes multi-dimensional set quality explicitly optimizable at every stage. The method achieves the best overall performance across ten benchmarks while issuing fewer retrieval calls.
Critical strengths
- The paper is well-organized and easy to follow. The appendices are extensive and genuinely useful for understanding the method.
- The proposed three-level, nine-dimension rubric hierarchy is a thoughtful design. Carrying the same rubrics through silver-label generation, RL reward computation, and distillation hints ties everything together, and it clearly contributes to the strong performance.
- The proposed Adaptive Tutoring Optimization (ATO) is a novel design. The idea of matching hint specificity to rollout quality is simple, intuitive, yet effective.
- Extensive experiments demonstrate the effectiveness of the proposed method. It consistently outperforms existing setwise and pointwise rerankers.
Critical weaknesses
- The novelty is limited. AdaTutoRank builds on RubricRanker's rubric-as-reward formulation and essentially extends it with a richer rubric hierarchy and a token-level distillation advantage on top of the scalar RL reward.
- BGE’s parameter counts should be reported in Table 1. BGE-Reranker-Large is only 0.6B yet remains highly competitive, beating several setwise approaches on the deep research benchmarks. Given this, I suggest comparing against stronger LLM-based pointwise rerankers such as ProRank (0.5B and 1.5B) [1] and Qwen3-Reranker [2] to test whether the setwise formulation consistently outperforms them.
- An efficiency comparison between pointwise and setwise rerankers at the same model scale would be valuable — for example, Qwen3-Reranker-8B (pointwise) versus AdaTutoRank (8B) or other 8B setwise models. This would clarify the practical trade-offs between the two paradigms.
- Real-world applications typically retrieve 100–1000 candidates at the first retrieval stage, but this paper only tests the top-20. It would be interesting to see whether the method supports larger candidate pools and how performance changes as k increases. Such an experiment would significantly strengthen the work.
Reference:
[1] Li, X., Shakir, A., Huang, R., Lipp, J., Clavié, B., & Li, J. (2026, July). Prorank: Prompt warmup via reinforcement learning for small language models reranking. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 1026-1037).
Questions for authors
The candidate pool size is fixed at the top 20 across all main experiments. Would it support larger pools, say 500 or 1000 candidates? If not, what design changes (e.g., two-stage selection, chunking, or a longer-context backbone) would be needed to handle such scenarios?
General Response
Dear Area Chair and Reviewers,
We sincerely appreciate your time and insightful feedback. We are glad that reviewers R1, R2, and R3 recognized that our work tackles an important problem beyond relevance-based reranking (R1, R2), that the nine-dimension rubric hierarchy and Adaptive Tutoring Optimization (ATO) form a well-motivated and novel design (R1, R2, R3), that our evaluation is extensive across RAG and deep research (R1, R2, R3), and that the paper is clear and well-organized (R2, R3).
The main concerns were whether adaptive tutoring gives a reliable gain beyond fixed hints and other design choices, how AdaTutoRank compares with stronger pointwise rerankers under matched conditions, and several design details. We addressed each with new experiments and clarifications, summarized below. Point-by-point responses are posted under each review.
We warmly encourage you to review our responses. We hope the new results and clarifications address your concerns, and we are happy to clarify any remaining questions.
Best regards,
Authors of Submission 20037