- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to show that, when training a visual document retriever, a large vision-language teacher's main value is not rich signals like score distillation, attention maps, or generated descriptions, but a simple binary judgment of which mined "negatives" are actually relevant and should be excluded from contrastive training.
Critical strengths
The paper's strongest contribution is its teacher-free control suite (§5), which matches N on candidate exposure (four candidates per query) and rejection rate (TopK-PercPos at α=0.864 rejects the same 41.9%), so the comparison isolates which negatives the teacher picks rather than how many or how much compute is spent.
Critical weaknesses
Insufficient scope and thin statistical power.
The paper focuses on one student (ColQwen2.5-3B), one corpus, and one teacher family, with only 2–3 seeds per arm. It is enough to establish a consistent sign and ordering, but not precise effect sizes. The authors concede the seeds support "a consistent sign and ordering of the mean gains" but "not precise effect sizes," and some finer comparisons are fragile: N's +1.7 margin over the rate-matched α=0.95 rule on v3 crosses seed counts (three-seed N minus a two-seed control), and D's temperature was tuned on the evaluation suites.
Questions for authors
S is compared only against a flat, information-free teacher-free target. When the target is an informative one built from the miner's own scores over the same four candidates, does S still beat it?
2 comments
Thank you again for your review. As the discussion ends on Nov 18, we wanted to check whether our response addresses your concerns. We would be glad to answer any remaining questions.
Thank you. Your two concerns, scope and statistical power, and your question about an informative target shaped most of our new experiments.
Scope. See part 1 of our general comment to all reviewers. The full study now also runs on a second, ~12× smaller student (ModernVBERT, 250M): as a late-interaction model with all four signals and their teacher-free controls, and as a single-vector model. The ordering holds on every student. For both new students, N and S improve on the baseline on every suite at every seed. One training corpus and one teacher family remain, and we keep them as stated limitations.
Seeds and the v3 margin. You are right that the abstract's +1.7 on v3 subtracts a two-seed control (α=0.864) from three-seed N. At matched seeds the margin barely moves:
We will report matched-seed margins in §5. Every new arm has three seeds.
D's temperature. Choosing τ_d on in-domain v1 alone, instead of v2/v3, selects the same value, τ_d=0.05 (Table 6; v1: −0.3 / +0.4 / +0.2 at τ_d = 0.01 / 0.05 / 0.1). In ReAlign's own single-vector setting, with its published temperature (not tuned on our suites), D still gains about nothing (+0.8 / +0.2 / −0.1 / +0.2).
S vs. an informative target. See part 2 of our general comment to all reviewers.