acceptodds
Under review as a conference paper at ICLR 2027

What Transfers from a VLM Teacher? Comparing Supervision Signals for Visual Document Retrieval

Abstract

Visual document retrievers are trained contrastively: each query is matched to one page labelled relevant — the positive — and pushed away from negatives, pages presumed irrelevant. Recent methods distil a vision–language model (VLM) teacher into the retriever by enriching that positive, transferring the teacher's attention over it or a description of it. We ask whether the teacher is better spent on the other side, judging the candidates the retriever mines as negatives, which the label says nothing about. With student, data, optimizer and evaluation fixed, teacher-judged hard negatives and score distillation raise ViDoRe v2 nDCG@5 from 55.2 to 62.6 and 63.0; description alignment, as adapted here, gains 2.6 points and attention grounding nothing measurable. Against teacher-free rules that select four candidates from the same mined pool at identical training compute, the best of which is the positive-aware threshold current systems use, the teacher's judgement adds 4.1 points on v2 and 1.7 on v3. This is consistent with how incomplete the labels are. Annotators judge about two of a query's four top-ranked mined candidates relevant, none of them labelled, so training pushes the retriever away from relevant pages treated as negatives. What reaches the student is coarse: under a greedily decoded 0–100 rating prompt, 82% of the teacher's ratings come back at one end of the scale or the other, and a relevant/irrelevant partition keeps most of the distillation gain. A ten-annotator audit places the teacher within the range of variation among human annotators, and finds it reliable (83% precision) where a query has a single determinate answer. We will release the code, the teacher's 3.3M judgements and page descriptions, the mined pools, the human audit and the trained adapters.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
visual document retrieval, knowledge distillation, hard negative mining, vision-language models, late-interaction retrieval, false negatives, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NVdbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to show that, when training a visual document retriever, a large vision-language teacher's main value is not rich signals like score distillation, attention maps, or generated descriptions, but a simple binary judgment of which mined "negatives" are actually relevant and should be excluded from contrastive training.

Critical strengths

The paper's strongest contribution is its teacher-free control suite (§5), which matches N on candidate exposure (four candidates per query) and rejection rate (TopK-PercPos at α=0.864 rejects the same 41.9%), so the comparison isolates which negatives the teacher picks rather than how many or how much compute is spent.

Critical weaknesses

Insufficient scope and thin statistical power.

The paper focuses on one student (ColQwen2.5-3B), one corpus, and one teacher family, with only 2–3 seeds per arm. It is enough to establish a consistent sign and ordering, but not precise effect sizes. The authors concede the seeds support "a consistent sign and ordering of the mean gains" but "not precise effect sizes," and some finer comparisons are fragile: N's +1.7 margin over the rate-matched α=0.95 rule on v3 crosses seed counts (three-seed N minus a two-seed control), and D's temperature was tuned on the evaluation suites.

Questions for authors

S is compared only against a flat, information-free teacher-free target. When the target is an informative one built from the miner's own scores over the same four candidates, does S still beat it?

2 comments

Thank you. Your two concerns, scope and statistical power, and your question about an informative target shaped most of our new experiments.

Scope. See part 1 of our general comment to all reviewers. The full study now also runs on a second, ~12× smaller student (ModernVBERT, 250M): as a late-interaction model with all four signals and their teacher-free controls, and as a single-vector model. The ordering holds on every student. For both new students, N and S improve on the baseline on every suite at every seed. One training corpus and one teacher family remain, and we keep them as stated limitations.

Seeds and the v3 margin. You are right that the abstract's +1.7 on v3 subtracts a two-seed control (α=0.864) from three-seed N. At matched seeds the margin barely moves:

  • N − α=0.864 at its two seeds: +4.2 on v2 and +1.6 on v3 (v3 per seed: +1.8, +1.4).
  • N − α=0.95 at three matched seeds: +4.1 on v2 and +2.4 on v3 (v3 per seed: +2.7, +2.1, +2.5).

We will report matched-seed margins in §5. Every new arm has three seeds.

D's temperature. Choosing τ_d on in-domain v1 alone, instead of v2/v3, selects the same value, τ_d=0.05 (Table 6; v1: −0.3 / +0.4 / +0.2 at τ_d = 0.01 / 0.05 / 0.1). In ReAlign's own single-vector setting, with its published temperature (not tuned on our suites), D still gains about nothing (+0.8 / +0.2 / −0.1 / +0.2).

S vs. an informative target. See part 2 of our general comment to all reviewers.

  • S is better on v2, but on v1, v3 and jina the miner's own scores match or beat it.
  • The teacher's ratings add value on ViDoRe v2's report collections, for both late-interaction students but not the single-vector one.
  • We revise S's claim accordingly.
  • The teacher's advantage for negative selection (N) holds on all three students.

Thank you again for your review. As the discussion ends on Nov 18, we wanted to check whether our response addresses your concerns. We would be glad to answer any remaining questions.

Reviewer_D4DEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies four ways to transfer supervision from a vision-language teacher to a visual document retriever: teacher-selected hard negatives, relevance-score distillation, attention grounding, and description alignment. Using a ColQwen2.5-3B student and a shared training setup, it finds that judging mined candidate pages improves retrieval more than enriching the labelled positive page. Teacher-selected negatives and score distillation raise ViDoRe v2 nDCG@5 from 55.2 to 62.6 and 63.0, respectively. Additional experiments and a human audit suggest that incomplete relevance labels make hard-negative mining problematic, and that much of the teacher’s useful supervision consists of distinguishing relevant from irrelevant candidates.

Critical strengths
  1. The paper poses a relevant research question on which teacher supervision signals improve document retrieval (although I would recommend some references for the motivation - weakness 1).
  2. The approach is relevant and controlled. All the studies hold the student, training data, optimizer, and evaluation setup fixed, with seed-matched baselines. This shows how each parameter impacts the student.
  3. There are some interesting observations (supported by evidence) on teacher-guided negative selection. Consistent retrieval gains: Negative selection and score distillation improve performance across all twelve datasets in the two primary out-of-domain suites.
  4. The paper shows human validation of the work (this is important in the age of LLM judges). The study supports label-incompleteness findings with a blinded human audit and provides detailed implementation, cost, per-seed results, and limitations.

The paper is certainly very relevant and has some strengths but the authors need to address some key weaknesses listed in the section below.

Critical weaknesses
  1. The motivation coming from the line # 44-46 are not substantiated through studies or references. Authors are suggested to present either an ablation related to this or references that pose this as the problem related to existing methods. The paper is also trying to establish this point but it is good to have a reference that states/shows otherwise.

  2. The study uses one late-interaction student family and one training corpus. The baseline is also below the released ColQwen2.5 checkpoint’s published v2 performance, and that checkpoint is not evaluated on v2/v3 in the authors’ harness. Does the relative effectiveness of the four supervision signals persist across student backbones, model sizes, or retrieval architectures? This is particularly relevant because description alignment was originally designed for a single-vector student and is adapted here to a late-interaction model.

  3. As I understand from the loss equation L_S, the score-distillation method gives the student four candidate pages. The teacher scores their relevance, and the student learns to distribute its preference across those pages accordingly. Couple of questions need to be addressed here: 3.1. Does the relative effectiveness of the four supervision signals persist across student backbones, model sizes, or retrieval architectures? Is there an ablation related to this? 3.2. Why did you choose four candidates per query? How sensitive are retrieval performance and the frequency of constant-target rows to the candidate count? A comparison with smaller and, where feasible, larger candidate sets would clarify the quality–compute trade-off. I believe this ablation is missing.

  4. The paper convincingly shows that teacher-guided negative selection helps, and provides evidence that false negatives are common. However, it does not establish how much of the improvement is specifically caused by removing false negatives, rather than by changing negative difficulty or diversity. Have you analyzed which types of teacher-selected negatives contribute most to the student’s retrieval gains? E.g., visually similar pages with incorrect answers, topically related but non-answering pages, or unrelated pages? Comparisons controlling for candidate count and retrieval difficulty would help distinguish the benefit of removing false negatives from that of selecting more informative true negatives.

  5. I would like to understand about the total number of judgements and the number of queries. Is the human audit sufficiently powered and representative to support the conclusions about teacher reliability? Although it contains 400 judgments, these cover 200 query-page pairs from only 50 queries, and the label-incompleteness estimate uses 29 queries with complete usable views. Please justify the sample size and clarify which conclusions the reported confidence intervals support. Auditing deeper-ranked candidates actually selected as negatives would also help establish reliability across the training pool. How does teacher reliability vary across document types, such as forms, financial tables, scientific articles, and infographics, and across query types, such as factual lookup, numerical reasoning, and broad semantic questions? Given the observed difference between specific and broad queries, does the audit contain sufficient examples within these groups to support its conclusions, or would a larger stratified audit be needed?

  6. Writing style - As evidenced by LLM usage disclosure, the authors have used AI tools to edit the paper. However, sometimes human generated text is more narrative and easy to understand. The sentence structuring in the paper is sometimes convoluted and the reader has to re-read the text multiple times to understand the parallel constructs. Authors might want to address this.

Questions for authors

Questions are listed as part of the weaknesses.

4 comments

Thank you for a careful review. Three new experiments answer your questions (a second student, a rank-shuffle control, and the released checkpoint in our harness). Questions raised by several reviewers are answered in our general comment to all reviewers, posted on this page; we refer to it below where relevant.

W1: support for the motivation (lines 44–46). Agreed; the revision adds references and points to our own evidence in §6. Unlabelled relevant passages among mined hard negatives are well documented in text retrieval (Qu et al., 2021; Cai et al., 2022; Moreira et al., 2024; Thakur et al., 2025). In our own blinded audit, annotators judged on average 2.04 of the miner's top four candidates relevant, while the dataset labels none of them.

W2 and W3.1: other students, and the released checkpoint. On ModernVBERT (250M), late-interaction and single-vector, the ordering N, S ≫ D > A holds (see part 1 of our general comment). The single-vector student is the setting D was designed for, and using ReAlign's own temperature, D gains +0.8 / +0.2 / −0.1 / +0.2. In our harness, the released colqwen2.5-v0.2 scores 88.6 / 59.7 / 50.8 / 81.0 (published v2: 60.9); our baseline (88.2 / 55.2 / 47.4 / 79.0) is below it because we train from the backbone (§3). With N or S, our student exceeds it on all four suites (N +1.0 / +2.9 / +1.2 / +0.8; S +0.3 / +3.3 / +1.9 / +0.4). These go into App. D.

W3.2: why four candidates? Four matches N, which appends four teacher-filtered negatives per query, so S and N train on the same pages per step at the same cost. A sweep over K needs new 3B runs of about 115 GPU-hours each (Table 12), which did not fit in the discussion period. We will state K as a fixed design choice and its sensitivity as a limitation.

W4: removing false negatives, or choosing harder ones? We added the control you describe (ModernVBERT, 3 seeds). In the rank-shuffle arm, each query takes its own candidates at the ranks N chose for another, randomly paired query. Pooled over the training set, the negatives' ranks (their difficulty), count and compute are exactly N's. Only the per-query judgement is gone, so pages the teacher rates relevant come back as negatives.

v1v2v3jina
rank shuffle − baseline+0.2−0.8+1.2+0.8
N − rank shuffle+1.6+4.2+0.2+1.7

N's difficulty profile alone is worth about nothing on v2; the gain comes from choosing, per query, which candidates are safe. Because the teacher removes false negatives and picks the remaining negatives in one decision, this measures the judgement as a whole. We have not analyzed which types of negatives contribute most; it is a natural next step. The rank-shuffle control goes into §5.

W5: is the audit powered and representative? The audit has 400 judgements: 50 queries × 4 pages × 2 independent annotators. The incompleteness estimate uses the 30 queries shown without their labelled page (29 after "cannot judge" grades). Its size was set by our annotation budget, and it is large enough for the three conclusions we draw; each clears its 95% interval (bootstrapped by query):

  • the top four mined candidates hold more than one relevant page: 2.04 [1.46, 2.61];
  • most pages the teacher protects are relevant: 71% [59, 80];
  • the teacher is more reliable on specific than on broad queries: 83% [71, 92] (33 queries) vs. 42% [24, 58] (17).

The audit covers only the top four candidates, as our limitations already state, and it is too small to compare document types or finer query types (5–15 queries per group). Those questions would need a larger, stratified audit, which we will add to the extended limitations (appendix).

W6: writing. Thank you; we agree that some sentences are hard to follow. In the revision we rewrite about twenty of the densest sentences, mainly in the abstract and §4–8. We split long sentences and replace the parallel constructs with separate statements.

Thank you again for your review. As the discussion ends on Nov 18, we wanted to check whether our response addresses your concerns. We would be glad to answer any remaining questions.

The authors addressed some of my comments. I will keep my current rating.

Thank you for the helpful comments.

Reviewer_Fg7hOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper compares four teacher-derived supervision signals for a ColQwen2.5-3B visual document retriever: filtered hard negatives, score distillation, attention grounding, and description alignment. With a shared training recipe, the first two improve ViDoRe v2 nDCG@5 from 55.2 to 62.6 and 63.0, respectively. Teacher-free selection controls and a human audit support the importance of unlabelled relevant candidates. Further experiments examine the near-binary rating targets and the effects of constant-target rows.

Critical strengths
  1. The negative-selection comparison isolates a meaningful teacher contribution. Table 2 compares N against five teacher-free rules using four negatives and identical training compute. N exceeds the α=0.95 positive-aware control by 4.1 v2 points on average and wins at every matched seed (Table 11). This supports consulting the teacher beyond merely adding mined candidates.
  2. The human audit independently supports the label-incompleteness diagnosis. Among 29 audited queries with complete top-four observations, annotators identify 2.04 relevant candidates on average, with a query-bootstrap interval of [1.46, 2.61]. Blinding and the leave-one-annotator-out checks in Appendix K strengthen this evidence beyond the teacher’s own relevance ratings.
  3. The constant-target analysis identifies a consequential training behavior. With constant targets on 42.6% of queries, masking those rows improves v2 by 2.7 points over S across three seeds (Table 3). Although exploratory, this result, together with Appendix A’s weight control, shows why inspecting target distributions can materially improve distillation.
Critical weaknesses
  1. The ranking of supervision strategies remains conditional on their adaptations. AGREE reports a 2.95-point v2 gain under its own recipe, whereas this study obtains approximately zero; ReAlign’s cosine-based formulation is transferred to MaxSim. Appendices G–H appropriately acknowledge these differences and provide sweeps. Nevertheless, D is never tested on top of mined negatives, and N/S require 4.8× the training step cost (Appendix J). Thus, the study establishes a useful configuration-specific ordering, but leaves the practical choice of where to allocate teacher supervision incompletely resolved.
  2. S lacks an informative teacher-free control. The flat-target control establishes that the chosen relevance targets outperform uniform regularization, but does not establish the added value of the VLM over an informative target available without it. Distilling the miner’s existing scores over identical candidates would directly address this attribution. Appendix P explicitly acknowledges this missing control. Because S matches N’s headline gains while requiring the same expensive preparation and training, this uncertainty matters to the recommendation about using a teacher.
  3. The partition experiment does not isolate the value of graded scores. It changes both score granularity and the prevalence of uniform targets: the latter rises from 42.6% to 54.3% of rows (§7). Since masking constant targets itself improves v2 by 2.7 points, the 1.75-point gap between graded and partition targets at matched seeds cannot cleanly measure the value of gradation. The authors acknowledge this confound and the prompt-specific scope. A comparison with identical row masking would make the information-transfer interpretation substantially more convincing.
Questions for authors
  1. Does description alignment improve the N-perc baseline when both use the same mined negatives, and what is its incremental training cost?
  2. How does S compare with distilling the miner’s own scores over identical candidates, using a temperature appropriate to those scores?
  3. Does the graded-versus-partition performance gap persist when both arms mask exactly the same rows and use identical loss normalization?
3 comments

Thank you for three precise questions. We ran a dedicated experiment for each.

Q1 and W1: does D help on top of mined negatives, and at what cost?

It does but not much. Adding D to N-perc (α=0.95) helps a little on every benchmark, but N stays ahead on both primary benchmarks (3B student, 3 seeds each):

v1v2v3jina
N-perc90.058.549.682.0
N-perc + D90.259.850.782.2
N89.662.652.081.8

D is cheap to add in training (+1.5 GPU-h per three-epoch run, about 1% on top of N-perc) once its one-time ≈110 GPU-h description pass is done.

How A and D were adapted (W1). The MaxSim adaptation is not what holds D back: on our single-vector student (ReAlign's own setting), with its published temperature (τ_d=0.01, not tuned on our suites), D gains +0.8 / +0.2 / −0.1 / +0.2. A is again about zero on ModernVBERT (−0.2 / −0.1 / −0.1 / −0.6). We cannot rule out that AGREE's full recipe with its own data and schedule does better.

On where to spend the teacher (W1): the 4.8× step cost comes from the four extra candidate forward passes, which every rule that trains on mined candidates pays, including the teacher-free N-perc. Given a mined pool, the teacher's judgement costs one ~37 GPU-h scoring pass (Table 12) and adds +4.1 on v2 (+2.0 and +2.2 on the two new students). D on top adds ~1% training cost for +1.3 on v2. Without candidate mining, D is the cheapest teacher signal, but its gains are small (+2.6 on v2 for the 3B student, minimal for the smaller ones). We will state this recommendation in §9.

Q2 and W2: S vs. the miner's own scores. We ran this control on all three students; see part 2 of our general comment to all reviewers.

Q3 and W3: graded vs. partition with identical masking. Thank you for asking for this test; it changes our conclusion about the partition. We ran two arms, graded and partition, on ModernVBERT (late interaction), with three seeds each. Both zero the KL on exactly the same rows (those whose four candidates all fall on one side of the 0.5 partition) and average the loss over all rows, so the loss normalization is identical. They share seeds, data order and candidates; only the target on the remaining rows differs:

ModernVBERT, 3 seedsv1v2v3jina
baseline73.750.837.269.7
partition, masked73.550.738.269.9
graded, masked74.054.538.770.5

The gap is big. The graded target beats the partition on every benchmark: by 3.7 on v2 (3.5, 3.5 and 4.2 per seed) and by 0.5 on v3, positive at every seed on both. With identical masking, the partition adds nothing over the baseline on v2. So, as you suspected, the paper's comparison could not isolate the value of the gradation, and once it is isolated, the gradation matters.

One likely reason is how the partition treats the teacher's in-between ratings. Most of the teacher's ratings sit at the two ends of the scale, but some fall in between, such as 0.5, the middle of the scale. The partition counts a page rated 0.5 as fully relevant, while S's sharp target gives it almost no weight next to a page rated 1.0. Our audit found pages rated 0.5 relevant only 28% of the time (§8), so the partition teaches the student to rank these mostly irrelevant pages as high as the clearly relevant ones. Queries where the teacher gives a 0.5 are also mostly broad (69%, against 22% of the others; §8), which may be why the gap is largest on v2. This would also explain why the same 0.5 cut works for N: there, a page rated 0.5 is only kept out of the negatives, which costs little, whereas the partition pushes it up as a positive.

What changes in the paper: we withdraw the claim that a relevant/irrelevant partition keeps most of the distillation gain (abstract, introduction, §7). The description of the ratings as near-binary stays (82% are at 0 or 100), but §7 will report this masked comparison and state that the few in-between ratings carry information that matters for S.

Thanks for the additional experiments; they addressed my questions well. Updating the claims on S and partition based on the new controls also makes the conclusions much cleaner, so I've raised my Soundness score from 2 to 3. Since S's gains over S-miner are mostly limited to ViDoRe v2 on late-interaction models, and the setup is still restricted to a single training corpus and teacher family, I'm keeping my overall rating at 3.

Thank you for raising the soundness score, and for the questions that led to these experiments.

Comments on the paper

2 comments

To all reviewers: new experiments and shared questions

We thank the reviewers for their careful reading. We ran several new experiments and answer shared questions here. All new arms have three seeds. We report seed means against a seed-matched baseline (nDCG@5 ×100). As in the paper, v2 and v3 are our primary benchmarks.

1. Beyond one student and architecture (D4DE, NVdb). We repeated the study on ModernVBERT (250M, trained from its MLM-only base), as a late-interaction model with all four signals and as a single-vector model. ModernVBERT is English-only, so its v2 and v3 scores use English queries. Change vs. baseline, with each cell giving v1 / v2 / v3 / jina-vdr:

3B ColQwen2.5ModernVBERT, late interactionModernVBERT, single-vector
N+1.4 / +7.4 / +4.5 / +2.8+1.8 / +3.4 / +1.4 / +2.4+1.7 / +4.7 / +2.1 / +2.5
S+0.7 / +7.8 / +5.3 / +2.4+1.9 / +3.9 / +3.2 / +2.5+2.3 / +2.4 / +3.0 / +2.5
D+0.4 / +2.6 / +1.1 / +0.1+1.1 / +0.8 / +0.4 / 0.0+0.8 / +0.2 / −0.1 / +0.2
A+0.2 / −0.1 / −0.3 / 0.0−0.2 / −0.1 / −0.1 / −0.6n/a*

The ordering holds on every student: the candidate-level signals (N, S) are well ahead of the positive-level ones (D, A). On both new students, N and S beat the baseline on every suite at every seed; D and A do not (*A needs per-patch scores, which a single pooled vector lacks).

2. Teacher ratings vs. an informative teacher-free target (Fg7h Q2, NVdb). S-miner is S with the teacher's ratings replaced by the miner's own MaxSim scores over the same four candidates, at T=0.0178, which matches the entropy of the teacher's target.

S − S-minerv1v2v3jina
3B ColQwen2.5−1.1+4.6−0.3−1.0
ModernVBERT, late interaction−1.0+3.1−0.5−0.9
ModernVBERT, single-vector−0.7−1.0−0.2−0.7

S-miner is a strong teacher-free baseline. On the 3B student it beats the baseline by +1.9 / +3.2 / +5.6 / +3.4, and on v1, v3 and jina-vdr it matches or beats S for all three students. We will report it as S's control.

The teacher's ratings help on ViDoRe v2's report collections, for both late-interaction students but not the single-vector one. We ruled out the obvious explanations: v3 has about as many relevant pages per query as v2, counting partial relevance as full leaves v3's result unchanged, and no query type in v3 shows a v2-sized gain. A possible reason, which we could not test: v2's queries are largely non-extractive, relying on implicit links between query and page (Cui et al., 2026), while many of v3's ask for a specific fact or number. The teacher's judgements may help most with this kind of implicit relevance.

We also tried a mix of the two targets on ModernVBERT (late interaction): the teacher decides which pages count as relevant, and the miner's scores decide how the target is shared among them. It recovers only about a third of S's v2 lead (+1.2 of +3.1). So the teacher helps not only by filtering out irrelevant pages: most of its value comes from its ratings among the relevant ones. The masked partition test in our reply to the reviewer Fg7h points the same way: the teacher's graded ratings matter beyond its relevant/irrelevant split.

What changes. We will state that the teacher's edge over S-miner is specific to ViDoRe v2's report collections and to late-interaction students. The claim for teacher-judged negatives stands: N beats the best teacher-free rule (N-perc, α=0.95) on all three students, by +4.1 / +2.0 / +2.2 on v2.

We will upload a revised PDF by Sunday (Nov 15), with all changes marked in blue.

We have uploaded the revised PDF, with all changes in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.