acceptodds
Under review as a conference paper at ICLR 2027

BandRank: Interval-Anchored ListMLE for Threshold-Transferable Pointwise Reranking

Abstract

Pointwise neural rerankers are widely deployed for their inference efficiency, yet learning them effectively involves balancing within-query discrimination against cross-query score consistency. Hard pointwise targets discard distinctions between relevance grades, whereas listwise ranking objectives are shift-invariant and lack an absolute anchor, leading to cross-query scale drift that degrades global thresholding. We propose **BandRank**, an interval-anchored ListMLE training framework. BandRank couples ListMLE with a smooth two-sided softplus penalty defined over query-shared score bands, encouraging predictions to remain within grade-specific intervals while ListMLE governs the relative ordering. To evaluate cross-query score reliability, we examine threshold transfer using transferred balanced accuracy (tr. BA) under frozen validation cuts. On the MultiVENT 2.0 video benchmark, BandRank improves on the previous state of the art by 7.6 points in nDCG@5 and 5.1 points in Recall@5, while improving tr. BA over a pointwise BCE baseline by up to 6.1 points. Furthermore, BandRank transfers thresholds stably even under calibration-data scarcity, and the same objective and band layout carry over to visual document (MMDocIR) and text passage (TREC DL) reranking. Code is available at https://anonymous.4open.science/r/ICLR2027-13167-7F87.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Information Retrieval, Reranking, Multimodal Retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_j73tOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the lack of cross-query comparability in pointwise reranker scores. BandRank combines ListMLE with grade-dependent interval anchors. The goal is to preserve pointwise, single-pass inference while making fixed thresholds transferable across queries. The experiments cover video, visual-document, and text-passage reranking.

Critical strengths
  1. The problem is practically important, and the method preserves pointwise inference complexity.

  2. The paper includes a point-anchor comparison with matched target locations.

  3. The paper reports both judged-negative and unjudged-as-negative evaluations. It also openly discusses midpoint preference at finite beta, upper-boundary degradation, and limitations of the point-anchor control.

  4. The same objective and band layout are applied to separately trained video, visual-document, and text rerankers. This supports objective portability across modalities. The paper does not claim single-checkpoint cross-modal transfer.

Critical weaknesses
  1. The strongest MultiVENT lower-boundary result depends on an unjudged-as-negative proxy. The paper transparently labels this as an unjudged setting and also reports judged Grade-0 results. The concern is limited to the headline MultiVENT lower-boundary value. On judged negatives, BandRank reaches 70.4 tr.BA, below Qwen3-VL-Reranker-8B at 76.1. Under the unjudged-as-negative protocol, BandRank reaches 85.6, above Qwen at 82.9. The ranking changes under the proxy-label protocol. Thus, 85.6 should not be interpreted as balanced accuracy on verified negatives. The paper should estimate the false-negative rate in the unjudged pool or clearly limit the claim to this deployment proxy.

  2. The current ablation does not fully isolate the benefit of interval anchoring. For Grades 1 and 2, the standalone anchor has a midpoint preference at finite beta. Grades 0 and 3 use one-sided constraints. The full BandRank objective also contains ListMLE, so its optimum is determined by the balance between ranking pressure and anchor preference. Therefore, the midpoint property is not a theoretical contradiction. The unresolved issue is attribution. The reported advantage over a soft point anchor may come from interval structure, gradient shape, or effective regularization strength. The paper should include curvature- or regularization-matched point anchors and a true zero-penalty hinge-band control.

Questions for authors
  1. MMDocIR and TRECDL support objective portability, but they do not include the same level of matched BCE and ListMLE-only controls as MultiVENT.
  2. The annotation procedure and permutation consistency are described in detail, but annotation tokens, runtime, and API or GPU cost are not reported.
  3. The training configuration reports batch size, epochs, ZeRO-2, and gradient checkpointing, but not GPU model, wall-clock time, or total GPU-hours.
  4. The paper mentions manual inspection without reporting the sample size, annotator count, or agreement.
5 comments
AuthorsOpenReview ↗

Response to Reviewer j73t - Part 1 of 3

We sincerely thank the reviewer for the insightful feedback and refer to General Response (G1–G4) alongside the details below.

W1: Scoping the MultiVENT lower-boundary claim and estimating the unjudged FNR

Scoping the claim. We agree that 85.6% measures full-pool filtering under a deployment proxy, not balanced accuracy on verified negatives. As already stated in the manuscript (Main Results), zero-shot Qwen3-VL-Reranker-8B leads on verified judged negatives (76.1% vs. 70.4%); we make no superiority claim over Qwen on this boundary. The revised Abstract and Introduction explicitly distinguish both protocols against our controlled BCE baseline:

"On MultiVENT 2.0, BandRank improves lower-boundary tr. BA over a controlled pointwise BCE baseline by 2.1 points on verified judged negatives (70.4% vs. 68.3%) and by 6.1 points in full-pool candidate filtering under an unjudged-as-negative proxy (85.6% vs. 79.5%)."

Estimating the FNR (General Response G2). Our blind human audit (by 3 domain-familiar lab members strictly under the 4-level rubric) on sampled unjudged test candidates (table in General Response G2) finds:

  • Only 0.8% of randomly sampled unjudged top-100 items are relevant by majority vote (99.2% true negatives).
  • Even among each model's highest-scoring unjudged items, missed positives remain rare (5.0% for BandRank vs. 0.8% for Qwen3-VL-Reranker-8B), showing no bias in BandRank's favor (if anything, counting retrieved relevant videos as false positives slightly penalizes it relative to Qwen3-VL-Reranker-8B).
  • Across the full 245K pool under false-negative rate corrections (\rho \in \\{1\\%, 5\\%, 10\\%\\}) and per-query top-kk exclusions (k∈1,5k \in \\{1, 5\\}), BandRank stays above Qwen and BCE in every setting (e.g., ρ=5\rho=5\\%: 87.4 vs. 84.6 vs. 81.0).

Why the ordering vs. Qwen flips. First, MultiVENT has only 974 judged G0G_0 pairs across 2,546 queries (<0.4< 0.4 per query, 0.4% of the top-100 pool); such a small sample may introduce statistical bias. Second, this gap reflects a shift between Qwen's pretraining prior and MultiVENT training supervision rather than interval anchoring: fine-tuning Qwen3-VL-Reranker-8B itself on MultiVENT lowers judged tr. BA under every objective, including plain BCE (76.1 → 67.1 BCE / 68.8 ListMLE-only / 69.9 BandRank; nNnT W2a), with BandRank degrading least. We list this as a limitation. On benchmarks with denser or exhaustive negative judgments (TREC DL with ~25 judged G0G_0/query and 100%-judged MMDocIR; General Response G3), BandRank outperforms Qwen3-Reranker-8B (+1.8 tr. BA, 83.8 vs. 82.0; +3.1 [email protected]) and Qwen3-VL-Reranker-8B (+1.6 / +1.8 tr. BA, 82.7 / 83.2 vs. 81.1 / 81.4; +15.1 / +15.5 [email protected]) directly on confirmed negatives.

AuthorsOpenReview ↗

Response to Reviewer j73t - Part 2 of 3

W2: Interval structure vs. gradient curvature vs. regularization strength

We disentangle these three factors via our hyperparameter sweep (Appendix D.1, revised Table 10) and the new mechanism controls in General Response G1 (Rows 4a–4c vs. Rows 3d & 5):

  1. Regularization strength (Appendix D.1, Table 10a): Across an 8×8\times weight range λAnchor∈[0.25,2.0]\lambda_{\text{Anchor}} \in [0.25, 2.0], BandRank remains highly stable (65.0–65.365.0\text{–}65.3 nDCG@5; 70.1–70.670.1\text{–}70.6 judged and 85.2–85.685.2\text{–}85.6 unjudged tr. BA), outperforming ListMLE + Soft-BCE (G1 Row 3d: 63.263.2 nDCG@5; 68.9/83.368.9 / 83.3 tr. BA) at every weight. Thus, BandRank's results do not hinge on tuning its anchor weight. Conversely, raising the point anchor's own strength to BandRank's level is exactly Row 4a below (per-grade reweighting), which serves as our regularization-matched point control.
  2. Curvature-matched point anchors (G1, Rows 4a–4b): We test (i) Curvature-matched BCE point (Row 4a), which reweights Eq. 4 per grade (1.92×1.92\times for G1G_1, 2.29×2.29\times for G2G_2) to match BandRank's restoring curvature at mgm_g (0.480 / 0.471 vs. unscaled 0.250 / 0.206), and (ii) Curvature-matched softplus point (Row 4b), which collapses the G1/G2G_1/G_2 bands to mgm_g within the logit-space softplus family (κ1=0.96,κ2=0.94\kappa_1=0.96, \kappa_2=0.94). Both eliminate the unscaled Soft-BCE ranking drop (63.2 → 65.2 nDCG@5) and reach 70.2 / 85.3 (Row 4a) and 71.2 / 85.5 (Row 4b) tr. BA.
  3. True zero-penalty hinge band (β→∞\beta\to\infty; G1, Row 4c): Extending our β∈0.5,1,2\beta \in \\{0.5, 1, 2\\} sweep (Table 10b: 70.2–70.9 / 85.1–85.6 tr. BA; 65.1–65.5 nDCG@5) to the β→∞\beta\to\infty limit, where the gradient is strictly zero inside [ℓg,ug][\ell_g, u_g], achieves the transfer gain on its own (70.5 / 85.5 tr. BA; highest untuned unjudged [email protected] of 85.2), confirming that transfer does not rely on finite-β\beta midpoint pull; β=1\beta=1 smoothing adds +0.9 nDCG@5 (64.4 → 65.3).

We agree with the reviewer: the gap to the unscaled soft point anchor is due to restoring strength, not interval width. Soft-BCE has ∼2×\sim 2\times weaker curvature at mgm_g, letting ListMLE drag G2G_2 downward (τ∗=0.429\tau^*=0.429); once curvature is matched, a point anchor performs on par with BandRank (Row 4b vs. Row 5, paired query-bootstrap 95% CIs of the difference: nDCG@5 [−0.3,+0.2][-0.3, +0.2], judged tr. BA [−0.9,+2.5][-0.9, +2.5], unjudged [−0.9,+0.5][-0.9, +0.5]), and Row 4b is exactly the zero-width member of our softplus family. Conversely, Row 4c shows midpoint pull is not needed either. What matters is a query-shared, grade-conditional anchor with sufficient bounded restoring force in logit space; the interval provides this without per-grade reweighting and is robust across width, β\beta, and λAnchor\lambda_{\text{Anchor}}. We have revised the mechanism discussion in the manuscript accordingly.

Q1: Matched BCE and ListMLE-only controls on MMDocIR and TREC DL

Matched same-backbone Pointwise BCE and ListMLE-only controls on TREC DL and MMDocIR (General Response G3) confirm that the MultiVENT pattern holds across every metric: BandRank improves ranking over BCE (+4.2 nDCG@10 on TREC DL; +1.0 to +2.4 R@3 on MMDocIR), improves confirmed-negative tr. BA over BCE (+5.3 and +2.6 / +0.5), ListMLE + Point (+1.3 and +2.7 / +2.0), and ListMLE-only (+1.7 and +3.7 / +2.7), and boosts untuned [email protected] over ListMLE-only by +24.2 on TREC DL and +17.5 / +10.0 on MMDocIR.

Q2 & Q3: Annotation tokens/cost and training hardware/GPU-hours

  • Annotation cost (Q2): All annotations were generated locally with open-weight Qwen3.6-27B on one 8× NVIDIA H20 (96 GB) node at 0 API cost (72–224 GPU-h per benchmark). Ground-truth positives are set to G3G_3 without VLM calls, and Stage 2 runs only on multi-candidate windows (≥2\ge 2 items) in G1..3G_{1..3} (for MultiVENT, Stage 1 averages 2.8K input / 95 output tokens; Stage 2 averages 9.2K input / 300 output tokens):
BenchmarkStage-1 callsStage-2 calls (windows × perms)Input / output tokensWall-clock (H20 GPU-h)
MultiVENT 2.0 (video, 32 frames)89,37949,130 (9,826 × 5)698M / 23.2M28 h (224 GPU-h)
MMDocIR (page image, ~768 vis. tok)514,70191,218 (30,406 × 3)790M / 65.0M24 h (192 GPU-h)
TREC DL (text passage)380,143126,000 (42,000 × 3)260M / 52.0M9 h (72 GPU-h)
  • Training hardware & GPU-hours (Q3): Fine-tuning for 2 epochs on one 8× NVIDIA H20 (96 GB) node takes 12.7 h (101.6 GPU-h, 560 steps) on MultiVENT 2.0, 20.8 h (166.4 GPU-h, 4,062 steps) on MMDocIR, and 4.9 h (39.2 GPU-h, 1,256 steps over the 20,088 lists with ≥2\ge 2 passages) on TREC DL.
AuthorsOpenReview ↗

Response to Reviewer j73t - Part 3 of 3

Q4: Sample size, annotator count, and agreement for the training-label audit

The "manual inspection" in the original appendix was an informal check by the authors, with no agreement statistics; we agree this was insufficient and replace it with the audits below. To substantiate our rule that demotes non-golden VLM Grade-3 predictions to Grade 2, we expanded our analysis in the revised manuscript with two quantitative validations (General Response G4):

  1. Official test-set ground truth (n=103n=103): To verify this G3→G2G_3 \to G_2 demotion rule on the 16,116 official human-judged MultiVENT test pairs, we checked all 103 non-positive candidates (official human Grade ≤2\le 2) that Qwen3.6-27B predicted as Grade 3: 89.3% (92/103) are indeed official Grade 2 (hard negatives), vs. 2.9% (3) Grade 1 and 7.8% (8) Grade 0.
  2. Blind 3-annotator audit on the training pool: Three domain-familiar lab members blindly graded 200 training pairs (50 random queries × 4 candidates: golden positive, non-golden VLM-G3G_3, and VLM-G2/1/0G_{2/1/0}) strictly under the 4-level rubric, plus 150 same-query pairs for pairwise preference:
    • Inter-annotator & human–VLM agreement: Pairwise order agreement is high—annotators agree with each other on 87.8–91.1% of non-tied pairs and with the training grade order on 78.1–81.9% (96.2–97.3% on golden ≻\succ VLM-G0G_0, 90.5% on G3≻G2G_3 \succ G_2, and 78.6% on G1≻G0G_1 \succ G_0). Residual human disagreement concentrates between adjacent intermediate negative tiers while same-grade pairs are mostly judged as ties—directly supporting BandRank's design of supervising grades via grade-level anchoring and relative orders rather than exact scalar regression, and motivating our tie-aware ablation (Reviewer nNnT, W3).
    • Validating G3→G2G_3 \to G_2 demotion on training data (n=44n=44): Among the 44 non-golden VLM-G3G_3 training candidates graded by all 3 annotators, Grade 2 is the most frequent majority label (73%, 32/44), vs. 11% Grade 3, 11% Grade 1, 5% Grade 0; in a 30-item factual checklist (89 ratings), 52.8% (47/89) show the requested scene/action type, while only 12.4% (11/89) depict the exact target event occurrence, confirming that they are primarily visually deceptive hard negatives (Grade 2).

Given the rebuttal time limit, we annotated 758 pairs in total across our 3-annotator audits (408 unjudged test + 350 training), and we will further expand this sample for broader statistical coverage in the next revision.

Reviewer_iXRQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Pointwise rerankers can provide strong within-query rankings without producing absolute scores that support a single threshold across queries. ListMLE can exploit a graded target ordering, but its shift-invariant objective does not identify an absolute score origin.

BandRank combines ListMLE with a grade-conditional softplus anchor that encourages each relevance grade to sit in a predefined score band, while keeping pointwise inference and without joint decoding. ListMLE handles relative order, the anchor handles absolute location. Graded supervision is built from a frozen VLM in two stages: coarse tiering then within-tier ranking. Evaluation is on MultiVENT 2.0, MMDocIR, and TREC DL with ranking metrics and transferred balanced accuracy using a frozen development threshold.

Critical strengths

1. Direct evidence for cross-query threshold control

The paper identifies a property that ranking metrics miss and provides mechanism-level evidence. ListMLE is shift-invariant, L(z+c1)=L(z)\mathcal{L}(z+c\mathbf{1})=\mathcal{L}(z), so good within-query ranking does not imply a transferable threshold. With a frozen development threshold, BandRank moves the operating threshold from 0.137 (BCE) and 0.086 (ListMLE-only) to 0.516, near the target 0.5. Changing the band layout moves the usable region accordingly without hurting ranking.

2. Efficient design with thorough and candid evaluation

BandRank adds a grade-conditional softplus anchor to ListMLE, no architecture change, no joint decoding, and keeps pointwise inference. Evaluation is unusually broad: loss ablations, threshold sweeps, alternative band layout, low-data calibration, score distributions, hyperparameter sensitivity, language/event transfer, pool depth, query normalization, and three separately fine-tuned modalities. The paper also reports limiting results - weaker than Qwen3-VL-Reranker-8B on judged negatives, no gain over BCE at Grade-3 vs Grade-2, and only modest ranking gain over ListMLE-only - which strengthens the conclusion that the anchor controls threshold location while graded listwise supervision drives most of the ranking gain.

Critical weaknesses

1. Insufficient Isolation of the Interval Anchor’s Incremental Contribution

The source of the improvement is not fully isolated. The controlled BCE baseline uses hard binary labels, whereas BandRank uses four relevance grades and intra-grade ordering. The ablation shows most of the ranking gain comes from ListMLE-only, with a smaller incremental gain from the band anchor. The point-anchor comparison aligns target optima but does not control gradient shape or regularization strength. Same-backbone comparisons against representative scale-calibrated ranking objectives or stronger ordinal pointwise losses are missing.

2. Limited Evidence for General Threshold Transfer

The central threshold-transfer evidence has unresolved evaluation and uncertainty issues. The largest reported gain over BCE occurs when approximately 245K unjudged candidates are treated as negatives, although unjudged candidates may contain relevant items and may affect models differently. On confirmed judged negatives, BandRank improves over the controlled BCE baseline but is substantially below Qwen3-VL-Reranker-8B, and BandRank does not improve the adjacent Grade-3-versus-Grade-2 boundary over BCE.

Questions for authors
  1. Can the authors provide a same-backbone and same-supervision comparison against at least one representative training-time calibrated-ranking objective and a stronger ordinal pointwise baseline? In particular, I would like to distinguish the benefit of (1) retaining graded labels, (2) listwise supervision, and (3) interval anchoring. A comparison in which BCE/ordinal pointwise, ListMLE-only, a calibrated listwise objective, and BandRank all receive the same grade information would materially clarify the paper's incremental contribution.

  2. Could the authors more directly validate that the observed threshold gains arise from reduced cross-query score-location mismatch? For example, please report the across-query distribution of grade-conditional score centers, per-query optimal thresholds, or global-threshold regret. Relatedly, for the unjudged-negative evaluation, can the authors manually reassess a sample of high-scoring unjudged candidates for each model or provide a sensitivity analysis to plausible false-negative rates? These analyses would materially affect my confidence that the maximum reported improvement reflects better threshold transfer rather than the unjudged-as-negative convention or improved class separation alone.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iXRQ — Part 1 of 2

We sincerely thank the reviewer for the constructive feedback and refer to General Response (G1–G4) alongside the diagnostics below.

W1 & Q1: Same-backbone, same-supervision isolation of the interval anchor

Our controlled same-backbone, 4-level supervision ablation in General Response G1 isolates each factor in Q1 and shows that BandRank (65.3 nDCG@5, 70.4 / 85.6 tr. BA, 70.8 / 83.2 [email protected]) outperforms ordinal pointwise, unanchored listwise, and all prior calibrated listwise losses:

  1. Retaining 4-level graded labels (Row 0 → Rows 1a–1b): Replacing binary BCE (59.9 nDCG@5; 68.3 / 79.5 tr. BA) with 4-level ordinal pointwise losses lifts nDCG@5 by +4.0 to +4.1 (63.9 / 64.0) and tr. BA to 68.9 / 83.1 (Point Anchor Only) and 69.3 / 84.0 (Band Anchor Only).
  2. Listwise supervision (Rows 1a–1b → Row 2): ListMLE-only further lifts ranking (64.6 → 65.8 nDCG@1; 64.0 → 64.5 nDCG@5), but shift invariance causes scale drift (τ∗=0.086\tau^*=0.086), collapsing untuned [email protected] from 69.9 / 83.3 to 57.0 / 59.1.
  3. Grade-conditional anchoring vs. prior calibrated losses & mechanism controls (Rows 2–4c → Row 5): BandRank (Row 5) beats all prior calibrated listwise losses (Rows 3a–3d: 62.3–64.7 nDCG@5; Yan et al., 2022; Yu et al., 2025) and widens confirmed-negative tr. BA gains over ListMLE-only to +1.7 on TREC DL and +3.7 / +2.7 on MMDocIR (G3). Rows 4a–4c further sharpen the mechanism: unscaled Soft-BCE (Row 3d) lagged due to ∼2×\sim 2\times weaker curvature at mgm_g (matching curvature in Rows 4a–4b recovers 65.2 nDCG@5), while a strictly flat Zero-penalty hinge band (β→∞\beta\to\infty, Row 4c) reaches 70.5 / 85.5 tr. BA (85.2 [email protected]) with zero interior gradient—confirming that the anchor's contribution is a query-shared, grade-conditional anchor with sufficient restoring force, which the interval form provides robustly without per-grade reweighting.

W2: Evidence for general threshold transfer

  1. Unjudged-negative reliability & dense human judgments (General Response G2 & G3): Both our 3-annotator blind audit on MultiVENT (G2) and matched controls on densely judged TREC DL and MMDocIR (G3) confirm that BandRank's transfer gains hold on verified human negatives. In G2, random unjudged items are 99.2% true negatives, top-2 missed positives are ≤5.0\le 5.0\\% without bias favoring BandRank, and BandRank ranks #1 in Unjudged tr. BA across all FNR corrections (\rho \in \\{1\\%, 5\\%, 10\\%\\}) and top-kk exclusions (+5.8 to +6.7 tr. BA over BCE). In G3, BandRank beats matched BCE (+5.3 and +2.6 / +0.5 confirmed-negative tr. BA) and ListMLE-only (+1.7 and +3.7 / +2.7, 20/20 splits).
  2. Why zero-shot Qwen3-VL-Reranker-8B is higher on MultiVENT judged G0G_0: Zero-shot Qwen's higher Judged tr. BA is measured on only 974 judged G0G_0 pairs (<0.4< 0.4/query, prone to sampling noise) and is not retained by any objective fine-tuned on MultiVENT, including plain BCE—fine-tuning Qwen itself yields 67.1 / 68.8 / 69.9 (BCE / ListMLE-only / BandRank; nNnT W2a)—reflecting a shift between Qwen's pretraining prior and MultiVENT supervision rather than the BandRank loss. Under the same MultiVENT supervision, BandRank attains the best judged tr. BA from both initializations (70.4 vs. 65.8–69.9, G1 Rows 0–3d; 69.9 vs. 67.1 / 68.8), whereas zero-shot Qwen's conservative scoring (μˉ0=0.083,μˉ2=0.220,μˉ3=0.286;τ∗=0.125\bar\mu_0=0.083, \bar\mu_2=0.220, \bar\mu_3=0.286; \tau^*=0.125) collapses upper-tier G3/G2G_3/G_2 tr. BA (55.6%), untuned [email protected] (49.9%), and ranking (45.0 nDCG@5); on denser confirmed negatives (TREC DL, MMDocIR; G3), BandRank beats Qwen3 rerankers (+1.8 and +1.6 / +1.8 tr. BA).
  3. Why binary BCE is slightly higher on G3G_3 vs. G2G_2: Binary BCE (G3=1,G0..2=0G_3=1, G_{0..2}=0) dedicates 100% of its capacity to the G3G_3 cut at the cost of collapsing G2G_2 onto G0G_0 (d2v0=0.66d_{2v0}=0.66 vs. BandRank's 1.08), whereas BandRank balances all four tiers. BandRank stays within 0.70.7 points on MultiVENT G3/G2G_3/G_2 (66.8% vs. 67.5%, n.s.), beats all external, listwise, and unscaled point baselines in Tables 2–3, leads on G3G_3 vs. unjudged (90.1% vs. 88.3%) and TREC DL G3/G2G_3/G_2 (+7.0 over BCE, 64.4% vs. 57.4%), and pins τ∗=0.785\tau^*=0.785 (vs. BCE's 0.3510.351).
  4. Split & seed stability (General Response G3; nNnT Q1): BandRank is highly stable across dev/test splits and training seeds (SD≤0.2\text{SD} \le 0.2 across 3 seeds; table in nNnT Q1), beating BCE on 20/20 splits (MultiVENT, TREC DL) and every seed/metric, and beating ListMLE-only on 20/20 splits (TREC DL, MMDocIR) and across seeds.
AuthorsOpenReview ↗

Response to Reviewer iXRQ — Part 2 of 2

Q2: Direct validation of reduced cross-query score-location mismatch & unjudged false-negative sensitivity

(a) Global-threshold regret (Δ\Delta) vs. within-query class separation: We decompose tr. BA into within-query separation (Per-query Oracle BA ↑, using each test query's optimal τq∗\tau_q^*, averaged over queries containing both classes) minus cross-query threshold mismatch (Global-threshold regret Δ↓=Oracle BA−tr. BA\Delta \downarrow = \text{Oracle BA} - \text{tr. BA}), directly confirming that BandRank's gain over ListMLE-only comes from cutting cross-query regret Δ\Delta (−0.4 to −3.5 points) rather than altering within-query separation (+0.1 to +0.4):

Benchmark & BoundaryModeltr. BA ↑Per-query Oracle BA ↑Global-threshold regret Δ\Delta ↓
MultiVENT 2.0 (Judged)Pointwise BCE68.387.319.0
ListMLE-only69.988.318.4
BandRank70.488.418.0
MultiVENT 2.0 (Unjudged)Pointwise BCE79.589.610.1
ListMLE-only84.494.710.3
BandRank85.694.89.2
TREC DL 19+20Pointwise BCE78.588.39.8
ListMLE-only82.190.88.7
BandRank83.891.27.4
MMDocIR (DSEwiki-ss)ListMLE-only79.097.018.0
BandRank82.797.214.5
MMDocIR (ColQwen2)ListMLE-only80.597.016.5
BandRank83.297.214.0

(b) Across-query distribution of grade centers and optimal thresholds: Auditing per-query score distributions confirms that BandRank aligns grade centers with their target bands, centers per-query optimal cutoffs at τq∗=0.494±0.152\tau_q^* = 0.494 \pm 0.152 (matching 0.50.5), and reduces relative cross-query drift (G2G_2 SD / Gap ↓) by 42–51% across all three benchmarks. Across MultiVENT 2.0 test queries, we report per-query mean score μq,g\mu_{q,g} (Grade g center), inter-tier separation μˉ2−μˉ0\bar{\mu}_2 - \bar{\mu}_0 (G2−G0G_2 - G_0 Gap ↑), relative drift SD(μq,2)/(μˉ2−μˉ0)\text{SD}(\mu_{q,2}) / (\bar{\mu}_2 - \bar{\mu}_0) (G2G_2 SD / Gap ↓), and Per-query τq∗\tau_q^*:

ModelGrade 0 centerGrade 2 center ([0.6,0.8][0.6,0.8])Grade 3 center ([0.8,1][0.8,1])G2−G0G_2 - G_0 Gap ↑G2G_2 SD / Gap ↓Per-query τq∗\tau_q^*
Pointwise BCE0.174 ± 0.1470.295 ± 0.1990.551 ± 0.2860.1211.640.224 ± 0.153
ListMLE-only0.117 ± 0.1610.261 ± 0.2110.488 ± 0.2940.1441.460.185 ± 0.168
ListMLE + Point0.369 ± 0.2240.577 ± 0.2130.751 ± 0.2120.2081.020.441 ± 0.163
Band Anchor Only0.436 ± 0.2020.619 ± 0.1590.768 ± 0.1660.1830.870.502 ± 0.138
BandRank0.417 ± 0.2180.630 ± 0.1810.784 ± 0.1820.2130.850.494 ± 0.152

BandRank cuts G2G_2 SD / Gap by 42–51% across all three benchmarks (1.64 / 1.46 → 0.85 on MultiVENT; 0.56 → 0.28 on TREC DL; 0.93 → 0.45 on MMDocIR-DSE) while centering τq∗\tau_q^* at 0.494 ± 0.152 (vs. 0.185±0.1680.185 \pm 0.168 for ListMLE-only).

(c) High-scoring unjudged audit & FNR sensitivity (General Response G2): Our blind 3-annotator audit and FNR/top-kk controls in G2 confirm that missed positives are rare (≤0.8\le 0.8\\% random; ≤5.0\le 5.0\\% top-2, with no bias favoring BandRank) and that BandRank remains #1 across all FNR corrections (\rho \in \\{1\\%, 5\\%, 10\\%\\}) and top-kk exclusions—while also cutting Δ\Delta by 2.5–3.5 points on 100%-judged MMDocIR (zero unjudged false negatives; Q2a).

We have added all above ablations and per-query diagnostics to the revised manuscript.

Thank you for the clear and thorough response. The additional analyses address my main concerns, so I have raised my soundness score.

We are glad that our response addressed your main concerns. Many thanks for your valuable feedback and for increasing the soundness score!

Reviewer_nNnTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes BandRank, which combines ListMLE with grade-specific interval anchoring to improve within-query ranking and cross-query threshold transfer while retaining pointwise inference. Experiments span video, visual document, and text retrieval, but the current evidence does not sufficiently establish the advantage over existing calibration methods.

Critical strengths

The paper clearly distinguishes ranking quality from cross-query score usability and adopts a practically meaningful frozen-threshold evaluation protocol. The objective is simple and introduces no additional cross-candidate interaction at inference. Controlled ablations support its compatibility with ListMLE relative to the tested point-anchor alternative. Evaluation covers three modalities, with additional analyses of limited calibration data, language shifts, and candidate-pool depth.

Critical weaknesses

Insufficient comparison with directly relevant methods. Although the paper cites Scale Calibration of Deep Ranking Models and Explain then Rank, it lacks empirical comparisons under matched supervision and evaluation protocols. The incremental value of interval anchoring therefore remains unclear. Calibration benefits are not sufficiently separated from reranker training gains. The models are initialized from general-purpose checkpoints rather than existing strong rerankers. It remains untested whether BandRank can improve threshold transfer while preserving an already competitive ranker. Over ListMLE alone, the judged-negative tr. BA gain is only 0.5 percentage points. Strict ordering may introduce unsupported preferences. Multiple permutations and LCS aggregation mitigate positional bias but do not establish reliable preferences among equally relevant candidates. In particular, retrieval-order fallback for G0 introduces ordering constraints without relevance-based justification. Completion of candidates omitted by the LCS is also insufficiently specified. Benchmark choice and claim scope need stronger justification. The rationale for making MultiVENT the primary benchmark is unclear. The largest threshold-transfer gain relies on treating unjudged candidates as negatives, while performance on judged negatives remains below the Qwen reranker. Moreover, the finite-temperature anchor retains a midpoint preference, qualifying the interpretation of within-band flexibility.

Questions for authors
  1. Why is MultiVENT the primary benchmark? Do the conclusions hold with more complete human judgments and multiple independent training seeds?
  2. Have the authors considered objectives that support ties or partial orderings rather than requiring a strict total order? Forcing an LLM to order every candidate may introduce noisy preferences, particularly when candidates are similarly.
5 comments
AuthorsOpenReview ↗

Response to Reviewer nNnT — Part 1 of 3

We sincerely thank the reviewer for the constructive feedback. We label the four Weakness paragraphs W1–W4 and refer to General Response (G1–G4).

W1: Matched comparison with Yan et al. (2022) and Yu et al. (2025)

Under identical 4-level supervision, BandRank outperforms all three scale-calibration paradigms from Yan et al. (2022) and Yu et al. (2025) across both ranking (65.3 nDCG@5) and threshold transfer (70.4 / 85.6 tr. BA, winning 19–20/20 splits). We evaluated all losses from Yan et al. (2022)—which Explain then Rank (Yu et al., 2025) directly adopts, as its test-time per-candidate LLM explanation generation violates single-pass O(N)O(N) scoring—in General Response G1 (Rows 3a–3d vs. Row 5):

  • Calibrated Softmax (Row 3a): Adding dummy anchor exp⁡(0)=1\exp(0)=1 fixes optimal logits to zi∗=log⁡(yi/y0)z_i^* = \log(y_i/y_0) (2g−12^g-1 gains; linear gains score 60.8 nDCG@5, 66.5 / 81.3 tr. BA), collapsing same-grade items toward one ratio (G1/G2G_1/G_2 medians merge at 0.706) and hurting ranking (62.3 nDCG@5).
  • Reference-Candidate ListMLE (Row 3b): A single virtual reference at logit 00 (grade 1.5) cannot anchor four tiers simultaneously; G0/G1G_0/G_1 items push G2G_2 downward (median 0.378, τ∗=0.211\tau^*=0.211), dropping untuned [email protected] to 63.1 / 68.4.
  • Multi-Objective point regression (Rows 3c–3d): Combining listwise loss with per-grade MSE or Soft-BCE (or Softmax + LogLoss: 62.8 nDCG@5, 68.6 / 82.6 tr. BA) trails in ranking (62.8–63.3 nDCG@5); for Soft-BCE this gap stems from weak restoring curvature, not from the use of point targets (G1, Rows 4a–4b).

W2: Initializing from an existing strong reranker and the +0.5 judged gain over ListMLE-only

(a) Fine-tuning from a strong existing reranker: Fine-tuning Qwen3-VL-Reranker-8B under matched settings, BandRank achieves the best ranking (65.8 nDCG@5, +20.8 over zero-shot) and beats both +BCE (+2.8 / +6.6 tr. BA) and +ListMLE-only (+1.1 / +1.1 tr. BA) on 20/20 splits, while restoring τ∗=0.511\tau^*=0.511 (+20.0 / +33.1 [email protected] over zero-shot):

Initialization & ObjectivenDCG@5Judged tr. BAUnjudged tr. BAτ∗\tau^*[email protected] (Judged / Unjudged)
Qwen3-VL-Rer.-8B45.076.182.90.12549.9 / 50.4
+ Pointwise BCE61.667.178.80.12654.6 / 56.8
+ ListMLE-only65.568.884.30.02751.2 / 51.7
+ BandRank65.869.985.40.51169.9 / 83.5
  • Preserving the strong ranker on judged G0G_0: Fine-tuning on MultiVENT lowers zero-shot Qwen's Judged tr. BA (76.1) under every objective, including plain BCE; BandRank degrades least (−6.2 vs. −7.3 ListMLE-only / −9.0 BCE), preserving the strong ranker best among matched objectives though not fully retaining its advantage on the small judged-G0G_0 subset (see W4b).

(b) Why the judged tr. BA gain over ListMLE-only is +0.5 on MultiVENT, and where it becomes decisive: On MultiVENT's small judged-G0G_0 set (n=974n=974), tuning τ∗\tau^* on the 30% dev split largely absorbs ListMLE-only's global location shift (τ∗=0.086\tau^*=0.086; +0.5 judged vs. +1.2 unjudged), whereas BandRank's lead widens decisively under denser human judgments and zero/scarce calibration:

  1. Denser human judgments (General Response G3): Confirmed-negative tr. BA gains over ListMLE-only widen to +1.7 on TREC DL (83.8 vs. 82.1) and +3.7 / +2.7 on 100%-judged MMDocIR (82.7 vs. 79.0; 83.2 vs. 80.5, 20/20 splits).
  2. Zero or scarce calibration (Fig. 2a–b): Without dev tuning (τ=0.5,ndev=0\tau=0.5, n_{\text{dev}}=0), BandRank beats ListMLE-only by +10.0 to +24.2 points across all benchmarks (e.g., 70.8% vs. 57.0% on MultiVENT), and still leads at ndev=10n_{\text{dev}}=10 queries (69.2% vs. 68.4% ListMLE-only, 67.0% Band only, 66.5% BCE).
AuthorsOpenReview ↗

Response to Reviewer nNnT — Part 2 of 3

W3 & Q2: Objectives supporting ties/partial orderings, G0G_0 fallback, and LCS completion

Two-stage design & LCS handling: Candidates omitted from the 5-run LCS still retain valid coarse grades (G1/G2G_1/G_2) for band anchoring and cross-grade ordering (G3≻G2≻G1≻G0G_3 \succ G_2 \succ G_1 \succ G_0), and are appended after the same-grade LCS subset in retrieval order. Stage 1 assigns coarse grades (G0–G3G_0\text{–}G_3), and Stage 2 breaks ties only when a query has ≥2\ge 2 items in grade g∈1..3g \in \\{1..3\\} (≤5\le 5/window; G0G_0 bypasses Stage 2, and 55.3% of per-query G1..3G_{1..3} groups are singletons).

Partial-order / tie-aware Plackett–Luce ablation: Tie-aware ablations show that (1) coarse-grade ties alone capture most of BandRank's gains (+4.6 nDCG@5, +2.1 / +5.5 tr. BA over BCE), and (2) within-band ordering further lifts ranking to 65.3 nDCG@5 without cross-tier drift. We generalize ListMLE (Eq. 2) to partial orders by restricting each item's Plackett–Luce denominator to itself and strictly lower-ranked items, omitting ties (LAnchor\mathcal{L}_{\text{Anchor}} fixed):

MethodG1..3G_{1..3} Intra-Grade OrderG0G_0 Intra-Grade OrdernDCG@1nDCG@5Judged tr. BAUnjudged tr. BA[email protected] (Judged / Unjudged)
Pointwise BCE (Tables 1–2)——61.959.968.379.557.4 / 59.4
BandRank (Coarse-Grade Ties Only)All tied (cross-grade only)Tied65.464.570.485.070.7 / 83.8
BandRank (Unanimous LCS Only)5-run LCS pairs only (rest tied)Tied65.664.770.385.070.6 / 83.9
BandRank (Tied-G0)LCS + retrieval tie-breakerTied66.164.970.785.170.8 / 83.7
BandRank (Full, Ours)LCS + retrieval tie-breakerRetrieval66.465.370.485.670.8 / 83.2
  1. Core gains do not require forced total orders: Tying all same-grade items (Coarse-Grade Ties Only) already captures most of the gain over BCE (+4.6 nDCG@5, +2.1 / +5.5 tr. BA, +13.3 / +24.4 [email protected]) and beats Band Anchor Only (64.0 nDCG@5).
  2. Within-band ordering adds signal safely: Adding LCS pairs and retrieval tie-breaking monotonically lifts nDCG@1/5 to 66.4 / 65.3. Because LAnchor\mathcal{L}_{\text{Anchor}} has bounded gradients (≤1\le 1) and confines each item to its band, minor intra-grade noise cannot cause cross-tier drift (Tied-G0 reaches 70.7 judged tr. BA; Full maximizes full-pool ranking and filtering).
  3. G0G_0 retrieval-order fallback is not needed for threshold transfer: Tying all G0G_0 items (Tied-G0) matches or exceeds Full on verified negatives (70.7 vs. 70.4 judged tr. BA; 70.8 vs. 70.8 judged [email protected]); the fallback only adds +0.4 nDCG@5 and +0.5 unjudged tr. BA, so Tied-G0 is an equally valid choice with no unsupported G0G_0 preferences.

W4: Unjudged negatives, Qwen reranker comparison, and finite-β\beta midpoint preference

  • (a) Reliability of unjudged negatives (General Response G2): Our 3-annotator blind audit (408 pairs) confirms 99.2% true negatives among random unjudged items, ≤5.0\le 5.0\\% missed positives even at top-2 ranks (with no bias favoring BandRank), and BandRank ranking #1 across all FNR corrections (\rho \in \\{1\\%, 5\\%, 10\\%\\}) and top-kk exclusions (G2).
  • (b) Why Qwen3-VL-Reranker-8B is higher on MultiVENT judged G0G_0: Zero-shot Qwen's higher judged tr. BA is measured on only 974 judged G0G_0 pairs (<0.4< 0.4/query, prone to sampling noise); this advantage is lost under every objective when fine-tuning Qwen on MultiVENT, including plain BCE (76.1 → 67.1 / 68.8 / 69.9 for BCE / ListMLE-only / BandRank; W2a), reflecting a shift between Qwen's pretraining prior and MultiVENT supervision rather than the BandRank loss. Under the same supervision, BandRank attains the best judged tr. BA from both initializations (70.4 vs. 65.8–69.9, G1 Rows 0–3d; 69.9 vs. 67.1 / 68.8), while zero-shot Qwen trails in ranking (45.0 nDCG@5), upper-tier G3/G2G_3/G_2 tr. BA (55.6%), and untuned [email protected] (49.9%); on densely judged TREC DL and 100%-judged MMDocIR (G3), BandRank beats Qwen3 rerankers on confirmed negatives (+1.8 and +1.6 / +1.8 tr. BA).
  • (c) Finite-β\beta midpoint preference vs. interval boundaries (General Response G1, Rows 4a–4c): Threshold transfer does not rely on finite-β\beta midpoint pull: a strictly flat zero-penalty hinge band (β→∞\beta\to\infty, Row 4c; zero interior loss/gradient) retains the full transfer gain (70.5 / 85.5 tr. BA, 85.2 [email protected]), while β=1\beta=1 smoothing lifts nDCG@5 from 64.4 to 65.3. Unscaled Soft-BCE (Row 3d) lagged only due to ∼2×\sim 2\times weaker curvature at mgm_g (0.250/0.2060.250 / 0.206 vs. BandRank's 0.480/0.4710.480 / 0.471); matching curvature (Rows 4a–4b) recovers 65.2 nDCG@5 with τ∗∈[0.483,0.520]\tau^* \in [0.483, 0.520].
AuthorsOpenReview ↗

Response to Reviewer nNnT — Part 3 of 3

Q1: Why MultiVENT? Do conclusions hold with more complete judgments and multiple seeds?

  • Why MultiVENT: We use MultiVENT 2.0 as the primary benchmark because (1) video reranking is an under-explored setting, especially for score reliability, and (2) among the three benchmarks, it uniquely combines 4-level human grades with large query scale (2,546 queries) to test both boundaries (G2/G0G_2/G_0, G3/G2G_3/G_2) and cross-query threshold transfer—whereas TREC DL 19+20 has 4-level grades but only 97 queries (~29 dev/split), and MMDocIR has 1,658 queries but only binary labels.
  • More complete human judgments (General Response G3): On TREC DL (2,467 confirmed G0G_0) and 100%-judged MMDocIR, BandRank beats matched BCE, ListMLE + Point, and ListMLE-only across ranking, confirmed-negative tr. BA (+1.7 to +5.3), and untuned [email protected] (+10.0 to +24.2).
  • Multiple independent training seeds: Across 3 independent seeds (42 / 43 / 44) on MultiVENT 2.0, BandRank beats BCE on every seed and metric and beats ListMLE-only on every seed for nDCG@5, R@5, and unjudged tr. BA (+0.4 mean judged gain):
Objective (Seeds 42 / 43 / 44)nDCG@5R@5Judged tr. BAUnjudged tr. BA
Pointwise BCE60.1 ± 0.149.6 ± 0.168.5 ± 0.379.6 ± 0.4
ListMLE-only64.6 ± 0.153.1 ± 0.170.1 ± 0.484.3 ± 0.1
BandRank65.3 ± 0.153.7 ± 0.170.5 ± 0.285.6 ± 0.0

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response — Part 1 of 2

We sincerely thank all reviewers for their insightful feedback and for recognizing our strengths: (1) Well-posed formulation: decoupling within-query ranking from cross-query calibration under frozen thresholds (nNnT, iXRQ, j73t); (2) Simplicity & efficiency: architecture-free, pointwise O(N)O(N) inference (nNnT, iXRQ, j73t); (3) Mechanism validation: tracking the 0.5 threshold and band layout without hurting ranking (iXRQ), comparing band vs. point anchors under ListMLE (nNnT, j73t); and (4) Broad, candid evaluation: spanning three modalities, diverse shifts, and transparent limitations (nNnT, iXRQ, j73t). Below are reviewer-suggested experiments and human audits (G1–G4; all revisions are highlighted in blue in the updated PDF).

G1. Source of gains: Controlled ablations & prior calibration losses (nNnT W1/W4c; iXRQ W1/Q1; j73t W2)

BandRank (Row 5) outperforms all prior scale-calibration losses (Rows 3a–3d) in both ranking (65.3 nDCG@5) and threshold transfer (70.4 / 85.6 tr. BA, τ∗=0.516\tau^*=0.516), while our new mechanism controls (Rows 4a–4c) confirm that transfer relies on sufficient grade-level restoring force rather than midpoint pull. We added controlled same-backbone ablations under identical 4-level supervision (only Row 0 binarizes G0..2G_{0..2}; Rows 3a–3d cover both Yan et al., 2022 and Yu et al., 2025) alongside [email protected] (Judged / Unjudged)—zero-calibration BA at the untuned default τ=0.5\tau=0.5 (ndev=0n_{\text{dev}}=0):

#Category & Training ObjectivenDCG@1nDCG@5Judged tr. BAUnjudged tr. BAτ∗\tau^*[email protected] (Judged / Unjudged)
0Binary Pointwise: Pointwise BCE (G3=1,G0..2=0G_3=1, G_{0..2}=0)61.959.968.379.50.13757.4 / 59.4
1aOrdinal Pointwise (Point): 4-level Soft-BCE (Point Anchor Only)65.263.968.983.10.51169.7 / 81.4
1bOrdinal Pointwise (Band): Band Anchor Only64.664.069.384.00.53169.9 / 83.3
2Unanchored Listwise: ListMLE-only65.864.569.984.40.08657.0 / 59.1
3aCalibrated Listwise: Calibrated Softmax (Yan et al., 2022; Yu et al., 2025)63.362.365.881.00.60066.2 / 80.9
3bCalibrated Listwise: Reference-Candidate ListMLE (Yan et al., 2022)65.864.768.884.30.21163.1 / 68.4
3cCalibrated Listwise: Multi-Objective Softmax + MSE (Yan et al., 2022)63.463.369.183.20.53169.2 / 82.5
3dCalibrated Listwise: Multi-Objective ListMLE + Soft-BCE64.463.268.983.30.42968.9 / 77.6
4aMechanism Control: ListMLE + Curvature-matched BCE point66.465.270.285.30.52070.5 / 83.5
4bMechanism Control: ListMLE + Curvature-matched softplus point66.765.271.285.50.48370.9 / 83.1
4cMechanism Control: ListMLE + Zero-penalty hinge band (β→∞\beta\to\infty)65.264.470.585.50.52670.1 / 85.2
5BandRank (β=1\beta=1, Ours)66.465.370.485.60.51670.8 / 83.2
  1. 4-level grades (Row 0 → 1a–1b) & ListMLE (→ Row 2): Retaining 4-level grades lifts nDCG@5 by +4.1 even under pointwise training. ListMLE-only (Row 2) further improves ranking (64.5), but shift invariance lets scores drift (τ∗=0.086\tau^*=0.086), collapsing untuned [email protected] to 57.0 / 59.1.
  2. Prior calibration losses (Rows 3a–3d → Row 5): A single reference candidate cannot anchor four tiers (Row 3b: τ∗=0.211\tau^*=0.211), while scalar-target losses (Rows 3a, 3c, 3d) trail in ranking (62.3–63.3 nDCG@5). BandRank (Row 5) applies bounded grade-band anchoring, beating all prior calibration losses (Rows 3a–3d) in both ranking (65.3 nDCG@5) and transfer (70.4 / 85.6 tr. BA, τ∗=0.516\tau^*=0.516).
  3. Curvature & flat-interval controls (Rows 4a–4c): Unscaled Soft-BCE (Row 3d) lagged because its restoring curvature at midpoint mgm_g is ∼2×\sim 2\times weaker than BandRank's, letting ListMLE pull G2G_2 downward (τ∗=0.429\tau^*=0.429); matching midpoint curvature (Rows 4a–4b) eliminates this drift and recovers 65.2 nDCG@5. Conversely, extending our β∈0.5,1,2\beta \in \\{0.5, 1, 2\\} sweep to the strictly flat β→∞\beta\to\infty limit (Zero-penalty hinge band, Row 4c; zero interior loss/gradient) still achieves 70.5 / 85.5 tr. BA (85.2 [email protected]), proving transfer does not rely on midpoint pull.
AuthorsOpenReview ↗

General Response — Part 2 of 2

G2. Reliability of unjudged negatives: Human audit & sensitivity (nNnT W4a; iXRQ W2/Q2; j73t W1)

Our new 3-annotator blind audit (408 pairs) and full-pool sensitivity controls confirm that unjudged false negatives are rare (≤0.8\le 0.8\\% random; ≤5.0\le 5.0\\% top-ranked), introduce no bias favoring BandRank, and leave BandRank #1 across all FNR corrections and top-kk exclusions. Specifically, 3 domain-familiar lab members blindly graded 408 unique unjudged top-100 test pairs across 60 random queries under the 4-level rubric (per query: 2 random unjudged + top-2 highest-scoring unjudged from BandRank, Qwen3-VL-Reranker-8B, BCE, and ListMLE-only; 120 pairs/group):

  • Random unjudged (120 pairs): Only 0.8% (1/120) are relevant (≥G2\ge G_2) by majority vote (99.2% true negatives; 0% G3G_3).
  • Top-2 highest-scoring unjudged per model (120 pairs/model): Only 5.0% (6/120) for BandRank, BCE, and ListMLE-only, and 0.8% (1/120) for zero-shot Qwen are unlabeled positives (≥G2\ge G_2), with no bias favoring BandRank.

Across all 245K unjudged pairs (2,546 queries), we further evaluate Unjudged tr. BA after (i) assuming a fraction \rho \in \\{1\\%, 5\\%, 10\\%\\} of unjudged items are unlabeled positives and correcting TNR via TNRcorr=TNRobs−ρ(1−TPR)1−ρ\text{TNR}_{\text{corr}} = \frac{\text{TNR}_{\text{obs}} - \rho(1-\text{TPR})}{1-\rho}, and (ii) dropping each query's top-kk highest-scoring unjudged items:

ModelUnjudged tr. BAρ\rho=1%ρ\rho=5%ρ\rho=10%Drop top-1Drop top-5
Pointwise BCE79.579.881.082.879.981.6
Qwen3-VL-Rer.-8B82.983.284.686.583.384.9
ListMLE-only84.484.786.288.284.786.1
BandRank85.685.987.489.586.087.4

BandRank leads every column in Unjudged tr. BA (+5.8 to +6.7 over BCE; +2.5 to +3.0 over Qwen; +1.2 to +1.3 over ListMLE-only).

G3. Complete human judgments: TREC DL & MMDocIR controls (nNnT Q1; j73t Q1)

On densely or exhaustively human-judged TREC DL 19/20 (52.7% judged, 2,467 confirmed G0G_0) and MMDocIR (100% judged), BandRank achieves the best confirmed-negative tr. BA and [email protected] among all matched controls and Qwen3 rerankers:

TREC DL 19/20nDCG@10Judged tr. BA (G0G_0)[email protected] (All)
Pointwise BCE68.478.567.1
ListMLE-only72.382.163.6
ListMLE + Point72.482.586.7
Qwen3-Rer.-8B70.782.084.7
BandRank72.683.887.8
MMDocIRR@3 (DSE / ColQ.)R@5 (DSE / ColQ.)tr. BA (DSE / ColQ.)[email protected] (DSE / ColQ.)
Pointwise BCE84.5 / 83.688.6 / 87.880.1 / 82.778.7 / 78.2
ListMLE-only83.5 / 84.888.1 / 88.879.0 / 80.565.3 / 73.3
ListMLE + Point84.1 / 85.088.3 / 88.980.0 / 81.268.3 / 79.6
Qwen3-VL-Rer.-8B86.1 / 86.989.5 / 90.481.1 / 81.467.7 / 67.8
BandRank85.5 / 86.089.2 / 89.682.7 / 83.282.8 / 83.3

Judged tr. BA gains (TREC DL; MMDocIR DSE / ColQ.): +1.7; +3.7 / +2.7 over ListMLE-only, +1.3; +2.7 / +2.0 over ListMLE + Point, +1.8; +1.6 / +1.8 over Qwen3 rerankers; untuned [email protected] rises +10.0 to +24.2 over ListMLE-only. As MMDocIR has only binary labels, its tr. BA tests the single relevant-vs-rest cut binary BCE is trained on, so smaller margins over BCE there (+2.6 / +0.5) are expected.

G4. VLM training label quality: Human audit & official judgments (j73t Q4; nNnT W3/Q2)

Auditing VLM training labels against official test judgments (n=103n=103) and a 3-annotator blind training audit (350 pairs) confirms that 89.3% of demoted G3→G2G_3 \to G_2 items are true Grade-2 hard negatives and that human–VLM pairwise order agreement is strong (78.1–81.9%):

  • G3→G2G_3 \to G_2 demotion on official test labels (n=103n=103): Among all 103 non-positive test candidates (official human Grade ≤2\le 2) predicted as Grade 3 by Qwen3.6-27B, 89.3% (92/103) are official Grade 2 hard negatives (vs. 2.9% G1G_1, 7.8% G0G_0).
  • Blind 3-annotator training audit (200 graded + 150 pairwise training pairs): Pairwise order agreement is 87.8–91.1% inter-annotator on non-tied pairs and 78.1–81.9% with the training grade order (96.2–97.3% on golden ≻\succ VLM-G0G_0, 90.5% on G3≻G2G_3 \succ G_2, 78.6% on G1≻G0G_1 \succ G_0), while residual human disagreement concentrates between adjacent intermediate tiers and same-grade pairs are mostly judged as ties—supporting grade-level anchoring with relative orders over exact scalar regression (nNnT W3).

All new ablations, controls, and audits (G1–G4) are in the revised PDF, and we will further scale up the human audits in the next revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.