- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the lack of cross-query comparability in pointwise reranker scores. BandRank combines ListMLE with grade-dependent interval anchors. The goal is to preserve pointwise, single-pass inference while making fixed thresholds transferable across queries. The experiments cover video, visual-document, and text-passage reranking.
Critical strengths
-
The problem is practically important, and the method preserves pointwise inference complexity.
-
The paper includes a point-anchor comparison with matched target locations.
-
The paper reports both judged-negative and unjudged-as-negative evaluations. It also openly discusses midpoint preference at finite beta, upper-boundary degradation, and limitations of the point-anchor control.
-
The same objective and band layout are applied to separately trained video, visual-document, and text rerankers. This supports objective portability across modalities. The paper does not claim single-checkpoint cross-modal transfer.
Critical weaknesses
-
The strongest MultiVENT lower-boundary result depends on an unjudged-as-negative proxy. The paper transparently labels this as an unjudged setting and also reports judged Grade-0 results. The concern is limited to the headline MultiVENT lower-boundary value. On judged negatives, BandRank reaches
70.4tr.BA, below Qwen3-VL-Reranker-8B at76.1. Under the unjudged-as-negative protocol, BandRank reaches85.6, above Qwen at82.9. The ranking changes under the proxy-label protocol. Thus,85.6should not be interpreted as balanced accuracy on verified negatives. The paper should estimate the false-negative rate in the unjudged pool or clearly limit the claim to this deployment proxy. -
The current ablation does not fully isolate the benefit of interval anchoring. For Grades 1 and 2, the standalone anchor has a midpoint preference at finite beta. Grades 0 and 3 use one-sided constraints. The full BandRank objective also contains ListMLE, so its optimum is determined by the balance between ranking pressure and anchor preference. Therefore, the midpoint property is not a theoretical contradiction. The unresolved issue is attribution. The reported advantage over a soft point anchor may come from interval structure, gradient shape, or effective regularization strength. The paper should include curvature- or regularization-matched point anchors and a true zero-penalty hinge-band control.
Questions for authors
- MMDocIR and TRECDL support objective portability, but they do not include the same level of matched BCE and ListMLE-only controls as MultiVENT.
- The annotation procedure and permutation consistency are described in detail, but annotation tokens, runtime, and API or GPU cost are not reported.
- The training configuration reports batch size, epochs, ZeRO-2, and gradient checkpointing, but not GPU model, wall-clock time, or total GPU-hours.
- The paper mentions manual inspection without reporting the sample size, annotator count, or agreement.
5 comments
Response to Reviewer j73t - Part 2 of 3
W2: Interval structure vs. gradient curvature vs. regularization strength
We disentangle these three factors via our hyperparameter sweep (Appendix D.1, revised Table 10) and the new mechanism controls in General Response G1 (Rows 4a–4c vs. Rows 3d & 5):
- Regularization strength (Appendix D.1, Table 10a): Across an weight range , BandRank remains highly stable ( nDCG@5; judged and unjudged tr. BA), outperforming
ListMLE + Soft-BCE(G1 Row 3d: nDCG@5; tr. BA) at every weight. Thus, BandRank's results do not hinge on tuning its anchor weight. Conversely, raising the point anchor's own strength to BandRank's level is exactly Row 4a below (per-grade reweighting), which serves as our regularization-matched point control. - Curvature-matched point anchors (G1, Rows 4a–4b): We test (i) Curvature-matched BCE point (Row 4a), which reweights Eq. 4 per grade ( for , for ) to match BandRank's restoring curvature at (0.480 / 0.471 vs. unscaled 0.250 / 0.206), and (ii) Curvature-matched softplus point (Row 4b), which collapses the bands to within the logit-space softplus family (). Both eliminate the unscaled Soft-BCE ranking drop (63.2 → 65.2 nDCG@5) and reach 70.2 / 85.3 (Row 4a) and 71.2 / 85.5 (Row 4b) tr. BA.
- True zero-penalty hinge band (; G1, Row 4c): Extending our sweep (Table 10b: 70.2–70.9 / 85.1–85.6 tr. BA; 65.1–65.5 nDCG@5) to the limit, where the gradient is strictly zero inside , achieves the transfer gain on its own (70.5 / 85.5 tr. BA; highest untuned unjudged [email protected] of 85.2), confirming that transfer does not rely on finite- midpoint pull; smoothing adds +0.9 nDCG@5 (64.4 → 65.3).
We agree with the reviewer: the gap to the unscaled soft point anchor is due to restoring strength, not interval width. Soft-BCE has weaker curvature at , letting ListMLE drag downward (); once curvature is matched, a point anchor performs on par with BandRank (Row 4b vs. Row 5, paired query-bootstrap 95% CIs of the difference: nDCG@5 , judged tr. BA , unjudged ), and Row 4b is exactly the zero-width member of our softplus family. Conversely, Row 4c shows midpoint pull is not needed either. What matters is a query-shared, grade-conditional anchor with sufficient bounded restoring force in logit space; the interval provides this without per-grade reweighting and is robust across width, , and . We have revised the mechanism discussion in the manuscript accordingly.
Q1: Matched BCE and ListMLE-only controls on MMDocIR and TREC DL
Matched same-backbone Pointwise BCE and ListMLE-only controls on TREC DL and MMDocIR (General Response G3) confirm that the MultiVENT pattern holds across every metric: BandRank improves ranking over BCE (+4.2 nDCG@10 on TREC DL; +1.0 to +2.4 R@3 on MMDocIR), improves confirmed-negative tr. BA over BCE (+5.3 and +2.6 / +0.5), ListMLE + Point (+1.3 and +2.7 / +2.0), and ListMLE-only (+1.7 and +3.7 / +2.7), and boosts untuned [email protected] over ListMLE-only by +24.2 on TREC DL and +17.5 / +10.0 on MMDocIR.
Q2 & Q3: Annotation tokens/cost and training hardware/GPU-hours
- Annotation cost (Q2):
All annotations were generated locally with open-weight
Qwen3.6-27Bon one 8× NVIDIA H20 (96 GB) node at 0 API cost (72–224 GPU-h per benchmark). Ground-truth positives are set to without VLM calls, and Stage 2 runs only on multi-candidate windows ( items) in (for MultiVENT, Stage 1 averages 2.8K input / 95 output tokens; Stage 2 averages 9.2K input / 300 output tokens):
| Benchmark | Stage-1 calls | Stage-2 calls (windows × perms) | Input / output tokens | Wall-clock (H20 GPU-h) |
|---|---|---|---|---|
| MultiVENT 2.0 (video, 32 frames) | 89,379 | 49,130 (9,826 × 5) | 698M / 23.2M | 28 h (224 GPU-h) |
| MMDocIR (page image, ~768 vis. tok) | 514,701 | 91,218 (30,406 × 3) | 790M / 65.0M | 24 h (192 GPU-h) |
| TREC DL (text passage) | 380,143 | 126,000 (42,000 × 3) | 260M / 52.0M | 9 h (72 GPU-h) |
- Training hardware & GPU-hours (Q3): Fine-tuning for 2 epochs on one 8× NVIDIA H20 (96 GB) node takes 12.7 h (101.6 GPU-h, 560 steps) on MultiVENT 2.0, 20.8 h (166.4 GPU-h, 4,062 steps) on MMDocIR, and 4.9 h (39.2 GPU-h, 1,256 steps over the 20,088 lists with passages) on TREC DL.
Response to Reviewer j73t - Part 3 of 3
Q4: Sample size, annotator count, and agreement for the training-label audit
The "manual inspection" in the original appendix was an informal check by the authors, with no agreement statistics; we agree this was insufficient and replace it with the audits below. To substantiate our rule that demotes non-golden VLM Grade-3 predictions to Grade 2, we expanded our analysis in the revised manuscript with two quantitative validations (General Response G4):
- Official test-set ground truth (): To verify this demotion rule on the 16,116 official human-judged MultiVENT test pairs, we checked all 103 non-positive candidates (official human Grade ) that Qwen3.6-27B predicted as Grade 3: 89.3% (92/103) are indeed official Grade 2 (hard negatives), vs. 2.9% (3) Grade 1 and 7.8% (8) Grade 0.
- Blind 3-annotator audit on the training pool: Three domain-familiar lab members blindly graded 200 training pairs (50 random queries × 4 candidates: golden positive, non-golden VLM-, and VLM-) strictly under the 4-level rubric, plus 150 same-query pairs for pairwise preference:
- Inter-annotator & human–VLM agreement: Pairwise order agreement is high—annotators agree with each other on 87.8–91.1% of non-tied pairs and with the training grade order on 78.1–81.9% (96.2–97.3% on golden VLM-, 90.5% on , and 78.6% on ). Residual human disagreement concentrates between adjacent intermediate negative tiers while same-grade pairs are mostly judged as ties—directly supporting BandRank's design of supervising grades via grade-level anchoring and relative orders rather than exact scalar regression, and motivating our tie-aware ablation (Reviewer nNnT, W3).
- Validating demotion on training data (): Among the 44 non-golden VLM- training candidates graded by all 3 annotators, Grade 2 is the most frequent majority label (73%, 32/44), vs. 11% Grade 3, 11% Grade 1, 5% Grade 0; in a 30-item factual checklist (89 ratings), 52.8% (47/89) show the requested scene/action type, while only 12.4% (11/89) depict the exact target event occurrence, confirming that they are primarily visually deceptive hard negatives (Grade 2).
Given the rebuttal time limit, we annotated 758 pairs in total across our 3-annotator audits (408 unjudged test + 350 training), and we will further expand this sample for broader statistical coverage in the next revision.
Response to Reviewer j73t - Part 1 of 3
We sincerely thank the reviewer for the insightful feedback and refer to General Response (G1–G4) alongside the details below.
W1: Scoping the MultiVENT lower-boundary claim and estimating the unjudged FNR
Scoping the claim. We agree that 85.6% measures full-pool filtering under a deployment proxy, not balanced accuracy on verified negatives. As already stated in the manuscript (Main Results), zero-shot Qwen3-VL-Reranker-8B leads on verified judged negatives (76.1% vs. 70.4%); we make no superiority claim over Qwen on this boundary. The revised Abstract and Introduction explicitly distinguish both protocols against our controlled BCE baseline:
Estimating the FNR (General Response G2). Our blind human audit (by 3 domain-familiar lab members strictly under the 4-level rubric) on sampled unjudged test candidates (table in General Response G2) finds:
Why the ordering vs. Qwen flips. First, MultiVENT has only 974 judged G0 pairs across 2,546 queries (<0.4 per query, 0.4% of the top-100 pool); such a small sample may introduce statistical bias. Second, this gap reflects a shift between Qwen's pretraining prior and MultiVENT training supervision rather than interval anchoring: fine-tuning Qwen3-VL-Reranker-8B itself on MultiVENT lowers judged tr. BA under every objective, including plain BCE (76.1 → 67.1 BCE / 68.8 ListMLE-only / 69.9 BandRank; nNnT W2a), with BandRank degrading least. We list this as a limitation. On benchmarks with denser or exhaustive negative judgments (TREC DL with ~25 judged G0/query and 100%-judged MMDocIR; General Response G3), BandRank outperforms Qwen3-Reranker-8B (+1.8 tr. BA, 83.8 vs. 82.0; +3.1 [email protected]) and Qwen3-VL-Reranker-8B (+1.6 / +1.8 tr. BA, 82.7 / 83.2 vs. 81.1 / 81.4; +15.1 / +15.5 [email protected]) directly on confirmed negatives.