- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a method for simultaneous listwise re-ranking and filtering of documents. Filtering uses a new token to set the threshold and is jointly fine-tuned with the re-ranking objective.
Critical strengths
- The problem of relevance filtering (or defining a cutoff threshold for ranking) is mostly underexplored, and it's nice to see work that contributes to this space.
- The results are really strong compared to the provided baselines (though I have concerns about the soundness of the baselines; see below). Table 5 is critical in showing the direct effects (the w/o Gate Loss rows).
Critical weaknesses
- The system is compared against really weak baselines. Most importantly, they're missing experimental and prose comparisons with a conceptually similar paper. Beyond this, less naive truncation methods (like the ones covered in the related work) should have been considered. In short, too many of the conclusions hinge on comparisons against weak top-k and fixed thresholding baselines. As a related issue, the baselines also do not share the backbone model or the exact training procedure, but the controlled ablations in Table 5 partially make up for this, though they still would have benefited from stronger truncation baselines there too.
EDIT: I realized that I forgot to mention the paper, https://arxiv.org/abs/2402.02764, at WWW 2024
Questions for authors
- For the baselines that use a globally tuned score threshold, what was the average number of documents returned per query?
2 comments
Response to Reviewer xhoi -- Part 2/2
3. Average number of documents returned by global thresholds.
We have computed the requested statistics from the formal per-query outputs of the original paper models. Each value below is obtained by first averaging the retained count over queries within each task and then macro-averaging across the 13 tasks.
| Original paper model and selection rule | Mean documents retained |
|---|---|
| BGE-v2-m3 + training-calibrated threshold | 5.66 |
| Qwen3-0.6B + training-calibrated threshold | 4.60 |
| Qwen3-4B + training-calibrated threshold | 2.01 |
| Qwen3-8B + training-calibrated threshold | 2.31 |
| Jina-v3 + training-calibrated threshold | 3.12 |
| Gate scores + training-calibrated threshold | 0.55 |
| Gate + learned boundary | 12.97 |
For completeness, in the additional matched-training experiments, FIRST-style and Rank-only retain averages of 1.72 and 1.50 documents under their respective overall test-oracle global thresholds. The newly trained Gate model retains an average of 9.60 documents using its native boundary. These results are reported separately from those of the original paper models.
We will report retained counts together with precision, recall, and F1, rather than treating a larger retained set as an improvement by itself. Under the unified offline recomputation for the original paper models, Gate achieves an average F1 of 0.4174, compared with 0.3778 for Jina, the strongest global-threshold baseline. The difference is 0.0396, with a paired-bootstrap 95% confidence interval of [0.0327, 0.0461]. This supports an aggregate selection benefit, not a claim of superiority on every task.
We have also separated standard full-list nDCG from post-truncation selection metrics and will correct the original ranking-performance statements accordingly.
We hope that the completed matched-training comparisons, exhaustive global-threshold diagnostic, and retained-set statistics address the main concerns about baseline reliability and the sources of improvement, and provide useful evidence for reconsidering the paper.
Response to Reviewer xhoi -- Part 1/2
We thank the reviewer for recognizing the value of jointly studying reranking and relevance filtering, and for identifying the related WWW 2024 paper. We have added evidence addressing the three main concerns: comparison with prior work, baseline strength, and the number of documents actually retained.
1. Clarifying the relationship to GenRT.
GenRT is directly relevant prior work. We will add an explicit citation and methodological comparison and revise the original characterization of the literature.
GenRT uses an encoder--decoder architecture to generate the reranked list sequentially while making truncation decisions. GateReranker instead reads document identifier logits and a gate logit at the first prediction step and obtains the retained set by comparing them, without sequentially generating document identifiers (Xu et al., List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation, WWW 2024; arXiv:2402.02764).
The contribution should therefore be positioned as a specific first-step-logit formulation that learns a list-conditioned boundary in a shared output space, rather than as the first method to jointly model reranking and truncation. We have not completed a direct experimental comparison with GenRT and do not claim empirical superiority over it.
2. Completed matched training and more extensive tuning of truncation baselines.
We have completed training and evaluation of FIRST-style, Rank-only, and Gate on all 13 tasks. The three models share an initialization checkpoint, the same seven-task training mixture, and a fixed budget of 2,000 optimizer steps. We also verified that the evaluation queries, candidate documents, and relevance labels match across models.
FIRST-style removes the gate token and gate losses, while Rank-only retains the gate-aware input and removes only the gate losses. FIRST-style denotes a controlled first-step scoring implementation sharing our ranking objective, rather than a complete reproduction of the original FIRST recipe. The budget is matched in data and optimizer steps, not GPU-hours; actual training implementation details will be documented.
Overall standard nDCG@10 is 0.4247, 0.4630, and 0.4763 for FIRST-style, Rank-only, and Gate, respectively. For retained-set selection, we additionally allowed the post-hoc baselines to use test labels to select the best global threshold:
This oracle diagnostic exhaustively covers the distinct retained sets obtainable by globally thresholding the saved scores. Thus, within the current scoring implementation, the observed difference cannot simply be attributed to an insufficient global-threshold sweep.
The diagnostic applies to the saved sigmoid scores. Because the current implementation applies sigmoid in BF16 before conversion to floating-point values, saturation and ties may occur. We therefore do not interpret these results as upper bounds for arbitrary raw-logit representations or alternative calibration methods.
Similarly, sweeping fixed K over {1,2,3,5,10,20,100,150} gives best F1 scores of 0.2772 for FIRST-style and 0.3039 for Rank-only, compared with 0.3594 for Gate. In particular, the comparison between 0.3013 and 0.3594 holds the Gate model and its document scores fixed, isolating the inference-time selection rule.
These experiments strengthen the controlled evidence in the paper. We agree, however, that they serve a different purpose from direct comparisons with learned truncation methods such as GenRT, and will clearly distinguish the two.
Throughout these selection analyses, F1 is computed per query and macro-averaged across tasks, with recall conditioned on relevant documents in the BM25 top-150 candidate pool.