- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper focuses on retrieving long documents by exploiting their structure. It proposes a distillation method, StarataDistill, for fine-tuning single-vector embedding retrievers. With StrataDistill, teachers are fine-tuned (SFT) separately on each data source, along with one teacher fine-tuned on all sources pooled. The teachers and the student share the same backbone, unlike classical distillation, which generally relies on a larger or stronger teacher. Document structure is used only by the teachers: each teacher embeds every section of a document and scores a query–document pair by combining the full-document similarity with that of the best-matching section. No section-level annotations are needed, since section relevance is inferred by the teacher. The student never sees sections, but learns to reproduce the teachers' relevance scores over candidate documents and inference uses one vector per document. The paper frames its experiments around five clear research questions. It reports gains over SFT on all five datasets across four backbones, comparisons of native sections against fixed-size chunks, joint training with short documents (which the method does not hurt), and comparisons against distillation from larger 8B teachers.
Critical strengths
The distillation method proposed is simple, practical and consistently better. Unlike common distillation, it requires no stronger teacher and no section-level labels for the documents. The authors do a good job of clearly explaining their ablations and research questions and have set up appropriate experiments to answer each. Through these explorations, is clear that the gains obtained by StartaDistill do not come just from training longer, strengthening the claim of their method. In addition, Separately, the method leaves inference unchanged (one vector per document, same index and scoring), making it easy to adopt in existing dense-retrieval systems.
Critical weaknesses
I have two weaknesses to point out, although I do not consider them too critical.
-
The paper claims their method is structure-aware, however, it is not clear if the gain is coming from document structure. The student learns to match the teacher's scores, which in turn relies on a best-section score. Could the same gains be obtained from a teacher whose score is the full-document score? With source-specific teachers, this would amount to distilling several expert teachers into one student, which is a technique that is known to help, with or without section scores. With the shared teacher, which equals the student's initialization, it would simply anchor the student to its starting point.
-
The "long-document" claim is weak. If we ignore QASPER, the mean document length across the remaining four corpora is roughly 860 tokens (Table 1), which arguably is not that long. And to tie this to weakness 1 above, on the one dataset that is genuinely long (QASPER), a strong teacher with no section consideration beats StartaDistill (Table 4, although this comparison is confounded by teacher size). So teacher quality might matter more than structure.
Questions for authors
[suggestion] In the shared-teacher setting, given that the teacher equals the student's initialisation, it is difficult to separate the contribution of best-section term from simply anchoring the student to \theta_mix. Running a teacher that uses the full document score would help clarify this.
2 comments
(2/2) Response to Reviewer DbTJ
W2. Document Length and Teacher Quality on QASPER
W2: “The "long-document" claim is weak. If we ignore QASPER, the mean document length across the remaining four corpora is roughly 860 tokens (Table 1), which arguably is not that long. And to tie this to weakness 1 above, on the one dataset that is genuinely long (QASPER), a strong teacher with no section consideration beats StartaDistill (Table 4, although this comparison is confounded by teacher size). So teacher quality might matter more than structure.”
A: Your estimate is correct. Without QASPER, the mean is 854.6 KaLM tokens across the four remaining tasks. The mean for documents relevant to the test queries is 1,827.6 tokens on those same tasks. A page or help article is a natural unit to index and return, even when only one of its sections contains the answer. QASPER indexes whole papers rather than individual pages. The table gives both length measures for comparison.
| Collection | All documents | Documents relevant to test queries |
|---|---|---|
| MTEB retrieval | 208.3 | 222.9 |
| CMTEB retrieval | 179.5 | 244.6 |
| Our five tasks | 1,740.0 | 2,518.5 |
| Our tasks excluding QASPER | 854.6 | 1,827.6 |
| LongEmbed | 20,851.0 | 20,851.0 |
| LoCo, full benchmark | 8,782.1 | 8,687.6 |
The LoCo row gives descriptive statistics for the full benchmark, including QASPER. The transfer evaluation below excludes QASPER. Appendix A.1, lines 768–771 gives the length comparison for our four tasks without QASPER. Before adaptation, KaLM-Embedding-V2.5 scores 47.80 mean nDCG@10 on our five tasks. The KaLM technical report, Tables 3 and 4 reports retrieval scores of 55.00 on English MTEB v1 and 73.42 on Chinese MTEB v1 for the same model version. These are different tasks, not a controlled comparison of document length.
Teacher quality on QASPER. The student trained on full-document scores from Qwen3-Embedding-8B does outperform StrataDistill with KaLM teachers on QASPER: 30.59 vs. 28.83 nDCG@10. Across the five tasks, the corresponding means are 52.95 and 54.88 (Table 5 and lines 486–489). The QASPER result favors the larger teacher there, but the two settings differ in both teacher and scoring rule. W1 isolates the latter: with the same source-specific KaLM teachers, adding section scores raises the five-task mean from 53.72 to 54.88.
Transfer to longer documents. For each backbone, we trained the Mixed-task SFT baseline and both StrataDistill students on the five main tasks: ESG, MIRACL, MultiDoc2Dial, QASPER, and WixQA. We evaluated those same checkpoints on LongEmbed and on LoCo excluding QASPER, without fine-tuning on either benchmark. Original is the released backbone. The table gives nDCG@10 with gains over Mixed-task SFT in parentheses.
| Benchmark | Backbone | Original | Mixed-task SFT | StrataDistill: shared teacher | StrataDistill: source-specific teachers |
|---|---|---|---|---|---|
| LongEmbed | KaLM | 55.25 | 62.94 | 64.31 (+1.37) | 63.77 (+0.83) |
| LongEmbed | Jina | 45.47 | 55.49 | 56.33 (+0.84) | 56.12 (+0.63) |
| LongEmbed | GTE | 60.86 | 67.58 | 68.33 (+0.75) | 67.95 (+0.37) |
| LongEmbed | Youtu | 54.70 | 58.91 | 60.74 (+1.83) | 61.35 (+2.44) |
| LoCo excl. QASPER | KaLM | 72.27 | 76.15 | 77.42 (+1.27) | 77.44 (+1.29) |
| LoCo excl. QASPER | Jina | 63.09 | 66.96 | 67.34 (+0.38) | 67.55 (+0.59) |
| LoCo excl. QASPER | GTE | 72.76 | 78.00 | 78.76 (+0.76) | 78.47 (+0.47) |
| LoCo excl. QASPER | Youtu | 66.38 | 69.87 | 71.73 (+1.86) | 72.35 (+2.48) |
Both student configurations improve on all four backbones on both benchmarks. For example, the source-specific Youtu student gains 2.44 points on LongEmbed and 2.48 on LoCo excluding QASPER. These tests complement the page- and article-level tasks without requiring every document in the original collections to be long. Section 4.2, Table 3, lines 324–345 reports the transfer results.
(1/2) Response to Reviewer DbTJ
Dear Reviewer
DbTJ,Thank you for recognizing the practical value of keeping a single vector per document. Your suggestion helped us distinguish the effect of the teacher from the effect of the scores it provides. The changes in the revised manuscript are marked in blue.
W1 & Q1. Full-document-only Teacher Scores
A: Your suggestion separates two explanations: does a shared teacher simply keep the student near its initialization, and do source-specific teachers help just by consolidating several experts? We retained KL supervision in both controls and removed only the section score. The table gives five-task mean Recall@10 / nDCG@10 for the students.
For KaLM, the shared teacher is the student initialization. Full-document KD gives 52.14 nDCG@10, close to the starting value of 52.03. Adding the section score with that same teacher raises the score to 53.64. The second explanation also accounts for part of the gain: combining expert teachers reaches 53.72 even without local matching. But adding local scores to those same experts raises the result to 54.88. On all four backbones, the local term improves nDCG@10 by 1.17 to 2.07 points with a shared teacher and by 0.77 to 1.60 with source-specific teachers. Section 4.7, Table 6, lines 491–520 reports the controlled comparisons.