acceptodds
Under review as a conference paper at ICLR 2027

StrataDistill: Structure-Aware Distillation for Long-Document Retrieval

Abstract

Long-document retrieval often depends on evidence confined to a small part of a document. Single-vector embedding models encode documents without the query, making local relevance difficult to preserve amid unrelated content. We introduce StrataDistill, a distillation method that uses document structure to guide supervised fine-tuning (SFT) of embedding models. Frozen teachers combine full-document similarity with the strongest local match to construct relevance distributions over candidate documents. A shared student learns these distributions through full-document scoring alongside supervised contrastive learning. Using source-specific teachers consolidates supervision from separately adapted models in the same retriever. The method reuses existing retrieval data and document structure without query-to-section annotations, and also supports short documents without decomposition. Across five long-document datasets and four embedding backbones, StrataDistill with source-specific teachers improves macro-average nDCG@10 by 2.85 to 4.66 points over mixed-task SFT. Further analysis shows a smaller gap between full-document and local-view scoring, consistent with partial transfer of local relevance into document representations. Inference preserves the original index size and scoring procedure, storing one vector per document.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-Document Retrieval, Text Embeddings, Document Structure, Knowledge Distillation, Supervised Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DbTJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on retrieving long documents by exploiting their structure. It proposes a distillation method, StarataDistill, for fine-tuning single-vector embedding retrievers. With StrataDistill, teachers are fine-tuned (SFT) separately on each data source, along with one teacher fine-tuned on all sources pooled. The teachers and the student share the same backbone, unlike classical distillation, which generally relies on a larger or stronger teacher. Document structure is used only by the teachers: each teacher embeds every section of a document and scores a query–document pair by combining the full-document similarity with that of the best-matching section. No section-level annotations are needed, since section relevance is inferred by the teacher. The student never sees sections, but learns to reproduce the teachers' relevance scores over candidate documents and inference uses one vector per document. The paper frames its experiments around five clear research questions. It reports gains over SFT on all five datasets across four backbones, comparisons of native sections against fixed-size chunks, joint training with short documents (which the method does not hurt), and comparisons against distillation from larger 8B teachers.

Critical strengths

The distillation method proposed is simple, practical and consistently better. Unlike common distillation, it requires no stronger teacher and no section-level labels for the documents. The authors do a good job of clearly explaining their ablations and research questions and have set up appropriate experiments to answer each. Through these explorations, is clear that the gains obtained by StartaDistill do not come just from training longer, strengthening the claim of their method. In addition, Separately, the method leaves inference unchanged (one vector per document, same index and scoring), making it easy to adopt in existing dense-retrieval systems.

Critical weaknesses

I have two weaknesses to point out, although I do not consider them too critical.

  1. The paper claims their method is structure-aware, however, it is not clear if the gain is coming from document structure. The student learns to match the teacher's scores, which in turn relies on a best-section score. Could the same gains be obtained from a teacher whose score is the full-document score? With source-specific teachers, this would amount to distilling several expert teachers into one student, which is a technique that is known to help, with or without section scores. With the shared teacher, which equals the student's initialization, it would simply anchor the student to its starting point.

  2. The "long-document" claim is weak. If we ignore QASPER, the mean document length across the remaining four corpora is roughly 860 tokens (Table 1), which arguably is not that long. And to tie this to weakness 1 above, on the one dataset that is genuinely long (QASPER), a strong teacher with no section consideration beats StartaDistill (Table 4, although this comparison is confounded by teacher size). So teacher quality might matter more than structure.

Questions for authors

[suggestion] In the shared-teacher setting, given that the teacher equals the student's initialisation, it is difficult to separate the contribution of best-section term from simply anchoring the student to \theta_mix. Running a teacher that uses the full document score would help clarify this.

2 comments
AuthorsOpenReview ↗

(1/2) Response to Reviewer DbTJ

Dear Reviewer DbTJ,

Thank you for recognizing the practical value of keeping a single vector per document. Your suggestion helped us distinguish the effect of the teacher from the effect of the scores it provides. The changes in the revised manuscript are marked in blue.

W1 & Q1. Full-document-only Teacher Scores

W1: “The paper claims their method is structure-aware, however, it is not clear if the gain is coming from document structure. The student learns to match the teacher's scores, which in turn relies on a best-section score. Could the same gains be obtained from a teacher whose score is the full-document score? With source-specific teachers, this would amount to distilling several expert teachers into one student, which is a technique that is known to help, with or without section scores. With the shared teacher, which equals the student's initialization, it would simply anchor the student to its starting point.”

Q1: “[suggestion] In the shared-teacher setting, given that the teacher equals the student's initialisation, it is difficult to separate the contribution of best-section term from simply anchoring the student to θmix\theta_{mix}. Running a teacher that uses the full document score would help clarify this.”

A: Your suggestion separates two explanations: does a shared teacher simply keep the student near its initialization, and do source-specific teachers help just by consolidating several experts? We retained KL supervision in both controls and removed only the section score. The table gives five-task mean Recall@10 / nDCG@10 for the students.

BackboneMixed-task SFTShared: full onlyShared: full + sectionSource-specific: full onlySource-specific: full + section
KaLM66.74 / 52.0366.97 / 52.1468.06 / 53.6468.22 / 53.7269.08 / 54.88
Jina62.54 / 46.8362.51 / 46.7363.42 / 48.2563.74 / 48.9364.19 / 49.70
GTE65.12 / 50.0965.98 / 50.8566.85 / 52.0267.25 / 52.4968.00 / 53.36
Youtu63.58 / 48.8564.57 / 49.9466.50 / 52.0166.53 / 51.9167.80 / 53.51

For KaLM, the shared teacher is the student initialization. Full-document KD gives 52.14 nDCG@10, close to the starting value of 52.03. Adding the section score with that same teacher raises the score to 53.64. The second explanation also accounts for part of the gain: combining expert teachers reaches 53.72 even without local matching. But adding local scores to those same experts raises the result to 54.88. On all four backbones, the local term improves nDCG@10 by 1.17 to 2.07 points with a shared teacher and by 0.77 to 1.60 with source-specific teachers. Section 4.7, Table 6, lines 491–520 reports the controlled comparisons.

AuthorsOpenReview ↗

(2/2) Response to Reviewer DbTJ

W2. Document Length and Teacher Quality on QASPER

W2: “The "long-document" claim is weak. If we ignore QASPER, the mean document length across the remaining four corpora is roughly 860 tokens (Table 1), which arguably is not that long. And to tie this to weakness 1 above, on the one dataset that is genuinely long (QASPER), a strong teacher with no section consideration beats StartaDistill (Table 4, although this comparison is confounded by teacher size). So teacher quality might matter more than structure.”

A: Your estimate is correct. Without QASPER, the mean is 854.6 KaLM tokens across the four remaining tasks. The mean for documents relevant to the test queries is 1,827.6 tokens on those same tasks. A page or help article is a natural unit to index and return, even when only one of its sections contains the answer. QASPER indexes whole papers rather than individual pages. The table gives both length measures for comparison.

CollectionAll documentsDocuments relevant to test queries
MTEB retrieval208.3222.9
CMTEB retrieval179.5244.6
Our five tasks1,740.02,518.5
Our tasks excluding QASPER854.61,827.6
LongEmbed20,851.020,851.0
LoCo, full benchmark8,782.18,687.6

The LoCo row gives descriptive statistics for the full benchmark, including QASPER. The transfer evaluation below excludes QASPER. Appendix A.1, lines 768–771 gives the length comparison for our four tasks without QASPER. Before adaptation, KaLM-Embedding-V2.5 scores 47.80 mean nDCG@10 on our five tasks. The KaLM technical report, Tables 3 and 4 reports retrieval scores of 55.00 on English MTEB v1 and 73.42 on Chinese MTEB v1 for the same model version. These are different tasks, not a controlled comparison of document length.

Teacher quality on QASPER. The student trained on full-document scores from Qwen3-Embedding-8B does outperform StrataDistill with KaLM teachers on QASPER: 30.59 vs. 28.83 nDCG@10. Across the five tasks, the corresponding means are 52.95 and 54.88 (Table 5 and lines 486–489). The QASPER result favors the larger teacher there, but the two settings differ in both teacher and scoring rule. W1 isolates the latter: with the same source-specific KaLM teachers, adding section scores raises the five-task mean from 53.72 to 54.88.

Transfer to longer documents. For each backbone, we trained the Mixed-task SFT baseline and both StrataDistill students on the five main tasks: ESG, MIRACL, MultiDoc2Dial, QASPER, and WixQA. We evaluated those same checkpoints on LongEmbed and on LoCo excluding QASPER, without fine-tuning on either benchmark. Original is the released backbone. The table gives nDCG@10 with gains over Mixed-task SFT in parentheses.

BenchmarkBackboneOriginalMixed-task SFTStrataDistill: shared teacherStrataDistill: source-specific teachers
LongEmbedKaLM55.2562.9464.31 (+1.37)63.77 (+0.83)
LongEmbedJina45.4755.4956.33 (+0.84)56.12 (+0.63)
LongEmbedGTE60.8667.5868.33 (+0.75)67.95 (+0.37)
LongEmbedYoutu54.7058.9160.74 (+1.83)61.35 (+2.44)
LoCo excl. QASPERKaLM72.2776.1577.42 (+1.27)77.44 (+1.29)
LoCo excl. QASPERJina63.0966.9667.34 (+0.38)67.55 (+0.59)
LoCo excl. QASPERGTE72.7678.0078.76 (+0.76)78.47 (+0.47)
LoCo excl. QASPERYoutu66.3869.8771.73 (+1.86)72.35 (+2.48)

Both student configurations improve on all four backbones on both benchmarks. For example, the source-specific Youtu student gains 2.44 points on LongEmbed and 2.48 on LoCo excluding QASPER. These tests complement the page- and article-level tasks without requiring every document in the original collections to be long. Section 4.2, Table 3, lines 324–345 reports the transfer results.

Reviewer_vsDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StrataDistill, a distillation method for single-vector embedding models on long documents. A frozen teacher combines full-document similarity with the best section similarity, and the student learns the resulting distribution while keeping one vector per document at inference. The method improves over mixed-task SFT across five datasets and four backbones.

Critical strengths
  1. Continued SFT and the shared-teacher variant isolate the effect of the distillation signal, and the results are consistent across backbones.
Critical weaknesses
  1. The source of the gain is not identified. The section term is not ablated. Source-specific teachers are not tested without local views. FGD from AAAI 2024 follows a similar design and is not compared.
  2. The experiments rely on synthetic training queries and small corpora with single runs. LongEmbed and LoCo are cited but not used.
Questions for authors
  1. What are the results when the section term is removed from the teacher score and when source-specific teachers are used without local views?
  2. Is the improvement over mixed-task SFT also observed on LongEmbed or LoCo?
3 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer vsDq

Dear Reviewer vsDq,

Thank you for recognizing the gains across backbones and for suggesting a direct comparison with FGD. We address the four points below. The manuscript revisions are marked in blue.

W1 & Q1. Source of the Gain and Removal of Local Views

W1: “The source of the gain is not identified. The section term is not ablated. Source-specific teachers are not tested without local views.”

Q1: “What are the results when the section term is removed from the teacher score and when source-specific teachers are used without local views?”

A: Removing local views lowers both Recall@10 and nDCG@10 for all four backbones, including when we use a separate teacher for each data source. For every pair of runs, the student starts from the same Mixed-task SFT checkpoint. Both runs keep the contrastive and KL losses. The teacher uses either the full-document score alone or the combination of full-document and section scores. Each cell gives mean Recall@10 / nDCG@10 across the five tasks.

BackboneMixed-task SFTShared: full onlyShared: full + sectionSource-specific: full onlySource-specific: full + section
KaLM66.74 / 52.0366.97 / 52.1468.06 / 53.6468.22 / 53.7269.08 / 54.88
Jina62.54 / 46.8362.51 / 46.7363.42 / 48.2563.74 / 48.9364.19 / 49.70
GTE65.12 / 50.0965.98 / 50.8566.85 / 52.0267.25 / 52.4968.00 / 53.36
Youtu63.58 / 48.8564.57 / 49.9466.50 / 52.0166.53 / 51.9167.80 / 53.51

The controls separate teacher specialization from local matching. Even with full-document scores only, changing from a shared teacher to source-specific teachers improves nDCG@10 by 1.58 to 2.20 points. Adding the section score to those experts gives a further 0.77 to 1.60 points. The local term also helps a shared teacher, by 1.17 to 2.07 points. We report the controls in Section 4.7, Table 6, lines 491–520.

AuthorsOpenReview ↗

(2/3) Response to Reviewer vsDq

W2. Comparison with FGD

W2: “FGD from AAAI 2024 follows a similar design and is not compared.”

A: We clarify this distinction in Section 2. Both methods retain one vector per document at inference, but they use spans for different purposes. FGD uses contrastive learning for whole documents and distills cross-encoder scores for passages and sentences. StrataDistill uses the best section match together with full-document similarity to rank whole documents. The student learns this ranking using only full-document scores. In our early experiments, learning relevance separately at the document and section levels did not consistently improve full-document retrieval. This motivated the document-level target.

The original FGD study, Section 3.3 used RoBERTa-base and capped document inputs at 512 tokens. We could not obtain its training code, so we implemented the published method. We first tested our implementation on an MS MARCO subset with 2,048 training examples, 1,024 test examples, and 20,000 candidate documents.

SettingnDCG@10Δ vs. Matching Control
Stage 1 Contrastive Learning90.53—
Stage 2 No-distillation Control92.25—
Stage 2 FGD92.87+0.62

On this subset, our FGD implementation outperformed the matched Stage 2 control by 0.62 nDCG@10 points. We then adapted its pooling and span selection to the KaLM backbone and longer inputs. We compared methods on the same five tasks from the same Mixed-task SFT checkpoint. Here, L denotes document length.

MethodLocal ViewsTeacherAvg. Recall@10Avg. nDCG@10Δ nDCG@10 vs. SFT
Mixed-task SFT——66.7452.03—
FGDFixed 128 / 64 tokensR2ANKER 0.11B64.0748.74−3.29
FGDProportional L/4 / L/8R2ANKER 0.11B65.6651.93−0.10
FGDProportional L/4 / L/8Qwen3-Reranker 0.6B65.7051.57−0.46
FGDProportional L/4 / L/8Qwen3-Reranker 8B65.1651.23−0.80
StrataDistill, shared teacherNative sectionsSame-backbone teacher68.0653.64+1.61
StrataDistill, source-specific teachersNative sectionsSame-backbone teachers69.0854.88+2.85

Length-proportional spans improved on fixed spans in our implementation. The best FGD setting reached 51.93 nDCG@10, compared with 52.03 at the SFT starting point. Replacing R2ANKER with a larger reranker did not close that gap. These results describe our adaptation rather than an evaluation of an official checkpoint. The main design differences are:

Design choiceFGDStrataDistill
Role of local viewsLearn relevance at several granularitiesUse the best section match to score the whole document
Distillation targetExternal reranker scores for passages and sentencesDocument rankings from full-document and best-section scores
Teachers across datasetsOne external reranker in our adaptationOne shared teacher or one teacher per dataset
Retrieval at inferenceOne vector per documentOne vector per document
Change from the KaLM SFT baseline−0.10 for our best FGD adaptation+1.61 with a shared teacher and +2.85 with source-specific teachers

With the same KaLM backbone, StrataDistill reached 53.64 with a shared teacher and 54.88 with source-specific teachers. No external reranker was needed to produce either target. The separate 8B teacher comparison in Section 4.6 tests an alternative source of document-level supervision.

AuthorsOpenReview ↗

(3/3) Response to Reviewer vsDq

W3. Synthetic Queries, Corpus Scale, and Single Runs

W3: “The experiments rely on synthetic training queries and small corpora with single runs.”

A: We used Qwen3-Max to generate training queries from each target collection. The 0.85 cosine-similarity filter excludes near-duplicate queries but does not enforce disjoint documents. The table separates corpus size from positive-document overlap. The denominator is the number of distinct test-positive documents. The percentage is the share that also appear as training positives.

DatasetDocuments in the collectionTest-positive documents also used as training positives
ESG59,77125.85%
MIRACL100,00035.32%
MultiDoc2Dial48891.58%
QASPER1,58598.74%
WixQA6,22137.65%

MultiDoc2Dial and QASPER have fewer than 2,000 documents and high overlap. ESG and MIRACL contain 59,771 and 100,000 documents. Overlap is below 40% on both, as it is on WixQA. ESG test queries are also synthetic. These five tasks evaluate adaptation to a known collection. W4 below tests transfer beyond those collections. Appendix A.1 describes this distinction in lines 763–771.

For variation across training runs, we repeated source-specific KaLM distillation with three seeds. The displayed nDCG@10 scores span 0.02 points, with a sample standard deviation of about 0.01.

MetricSeed 42Seed 43Seed 44
nDCG@1054.8654.8854.88
Recall@1069.0869.0569.10

For variation across test queries, we computed paired-bootstrap intervals for the nDCG@10 gain of the source-specific student over Mixed-task SFT. The intervals below are for the mean across the five datasets, not for variation across seeds.

Macro-average Δ nDCG@10 over Mixed-task SFT [95% CI]KaLMJinaGTEYoutu
All Datasets+2.86 [+2.53, +3.20]+2.87 [+2.57, +3.19]+3.28 [+2.85, +3.70]+4.67 [+4.26, +5.07]

All four macro intervals and all 20 dataset-level intervals exclude zero for that comparison. The seed results and per-dataset intervals appear in Appendix C.4, lines 988–1006.

W4 & Q2. Evaluation on LongEmbed and LoCo

W4: “LongEmbed and LoCo are cited but not used.”

Q2: “Is the improvement over mixed-task SFT also observed on LongEmbed or LoCo?”

A: Yes. We trained the Mixed-task SFT baseline and both StrataDistill students on ESG, MIRACL, MultiDoc2Dial, QASPER, and WixQA. We then tested the same checkpoints on LongEmbed and on LoCo excluding QASPER, without fine-tuning on either benchmark. Original is the released backbone. The table gives nDCG@10, with gains over Mixed-task SFT in parentheses.

BenchmarkBackboneOriginalMixed-task SFTStrataDistill: shared teacherStrataDistill: source-specific teachers
LongEmbedKaLM55.2562.9464.31 (+1.37)63.77 (+0.83)
LongEmbedJina45.4755.4956.33 (+0.84)56.12 (+0.63)
LongEmbedGTE60.8667.5868.33 (+0.75)67.95 (+0.37)
LongEmbedYoutu54.7058.9160.74 (+1.83)61.35 (+2.44)
LoCo excl. QASPERKaLM72.2776.1577.42 (+1.27)77.44 (+1.29)
LoCo excl. QASPERJina63.0966.9667.34 (+0.38)67.55 (+0.59)
LoCo excl. QASPERGTE72.7678.0078.76 (+0.76)78.47 (+0.47)
LoCo excl. QASPERYoutu66.3869.8771.73 (+1.86)72.35 (+2.48)

The gain holds for both teacher choices in all eight combinations of benchmark and backbone. Taking the better configuration for each backbone gives gains of 0.75 to 2.44 points on LongEmbed and 0.59 to 2.48 on LoCo excluding QASPER. The result does not depend on selecting the better configuration: both student scores exceed the corresponding SFT score in every row. The revised paper reports these evaluations in Section 4.2, Table 3, lines 324–345.

Reviewer_UejmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors observe that in long documents, query-relevant information is often concentrated in a single section. However, single-vector embedding models compress the entire document into one vector regardless of the query, making it difficult to capture such localized relevance. To address this limitation, the paper proposes StrataDistill, a method that improves the model only during training.

  1. Teacher model: A checkpoint fine-tuned via SFT using the same backbone is kept frozen. For each candidate document, the teacher computes a score by averaging the document-level similarity and the maximum section-level similarity. Separate teachers can also be used for different datasets.
  2. Student model: The student scores documents using only document-level embeddings, learning to match the teacher's score distribution via a KL-divergence loss while jointly optimizing the conventional InfoNCE loss.
  3. Inference: Only the student model is used at inference time, requiring just one vector per document. Therefore, index size and retrieval costs remain unchanged. Across five datasets and four backbones, using dataset-specific teachers improves average nDCG@10 by 2.85–4.66 points compared with mixed-task SFT.
Critical strengths
  1. Consistent performance improvements without additional inference costs. The dataset-specific teacher variant improves both evaluation metrics across all 20 combinations (4 backbones x 5 datasets). The Continued SFT baseline (−0.42 to +0.82 points) rules out the explanation that the gains simply come from training for longer. Since no changes to the index or retrieval pipeline are required, the method is easy to deploy in practice.

  2. Analysis supporting the proposed mechanism. Even after SFT, section-level scores remain approximately 5 points higher than document-level scores, but this gap narrows after distillation. Since section-level performance remains nearly unchanged while direct retrieval performance improves, the results are consistent with the claim that localized relevance is partially transferred into document-level embeddings. The finding that a teacher of the same size outperforms an 8B embedding/reranker teacher is also practically interesting.

Critical weaknesses
  1. It is not established that the performance gains come from section-level information.

StrataDistill changes two things simultaneously compared with the baseline:

  • (a) It introduces KL-based soft-label distillation itself, including factors such as temperature, loss weighting, and the distribution of in-batch candidates.
  • (b) It incorporates the maximum section-level similarity into the teacher's score. The paper's Continued SFT baseline and the setting with kd=0 only control for additional InfoNCE training; they do not isolate the effect of (a). The essential missing experiment is self-distillation using only document-level scores, without section-level information. This is particularly important for the shared-teacher variant, where the teacher and student start from the same model, meaning that soft-label self-distillation alone could provide a regularization effect.

Furthermore, mixed-task SFT actually degrades performance for some backbones (Jina WixQA: 61.64->55.21; Youtu MIRACL: 73.30->69.86, with Youtu's average performance remaining nearly unchanged). Thus, some of the gains may reflect recovery from poorly tuned SFT rather than the proposed method itself. The paper also does not investigate different mixing ratios between document-level and section-level scores.

Moreover, the contribution of structure awareness, as implied by the title, appears limited. Using the original section structure yields only a 0.29-point improvement in nDCG@10 over fixed-length chunking (54.88 vs. 54.59), while fixed-length chunks retain 89.8% of the gains. This suggests that the results support the use of maximum chunk-level similarity as a teacher score more strongly than they support structure awareness per se. Since the 0.29-point difference is based on a single run, its statistical significance is also unknown.

  1. Limitations of the evaluation setup: the same corpus, synthetic queries, and relatively short documents.
  • All training queries are generated by Qwen3-Max from the very same document collections used for evaluation (e.g., 488 documents in MultiDoc2Dial and 1,585 in QASPER). Even the ESG test queries are synthetic. Consequently, the improvements may reflect adaptation to the specific corpus rather than better long-document retrieval in general. Deduplication using a cosine similarity threshold of 0.85 does not resolve this concern.
  • The method is not evaluated on established long-document retrieval benchmarks that are not used for training, such as LongEmbed, MLDR, LoCoV1, or MS MARCO Document.
  • Despite the emphasis on long documents, the average document lengths are only 479 tokens for WixQA and 747 tokens for ESG (roughly one page). QASPER appears to be the only dataset that genuinely involves long documents.
  • The paper reports neither repeated runs with different random seeds nor statistical significance tests. Many improvements are below one point (e.g., +0.08 on Jina MultiDoc2Dial), while the substantial gains from dataset-specific teachers on WixQA (+3.30 to +4.68) are measured on a test set of only 400 queries.
Questions for authors
  1. Self-distillation without section-level information: What are the results of self-distillation using only document-level scores, without section-level information?

  2. Training–evaluation corpus overlap and zero-shot generalization: Were the training queries generated from the same documents used in the test corpus?

6 comments
AuthorsOpenReview ↗

(1/4) Response to Reviewer Uejm

Dear Reviewer Uejm,

Thank you for recognizing the value of single-vector retrieval and suggesting the full-document control. Revisions are in blue.

W1 & Q1. Full-document-only Self-distillation

W1: “It is not established that the performance gains come from section-level information. StrataDistill changes two things simultaneously compared with the baseline: (a) It introduces KL-based soft-label distillation itself, including factors such as temperature, loss weighting, and the distribution of in-batch candidates. (b) It incorporates the maximum section-level similarity into the teacher's score. The paper's Continued SFT baseline and the setting with kd=0 only control for additional InfoNCE training; they do not isolate the effect of (a). The essential missing experiment is self-distillation using only document-level scores, without section-level information. This is particularly important for the shared-teacher variant, where the teacher and student start from the same model, meaning that soft-label self-distillation alone could provide a regularization effect.”

Q1: “Self-distillation without section-level information: What are the results of self-distillation using only document-level scores, without section-level information?”

A: With either teacher configuration, removing the section score lowers both retrieval metrics on all four backbones. We compared two students initialized from the same Mixed-task SFT checkpoint. They use the same frozen teacher, training examples, candidate pool, and contrastive and KL losses. Only the teacher scores change: one teacher target uses full documents alone, and the other also uses the best section match. Each cell gives mean Recall@10 / nDCG@10 over the five tasks.

BackboneMixed-task SFTShared: full onlyShared: full + sectionSource-specific: full onlySource-specific: full + section
KaLM66.74 / 52.0366.97 / 52.1468.06 / 53.6468.22 / 53.7269.08 / 54.88
Jina62.54 / 46.8362.51 / 46.7363.42 / 48.2563.74 / 48.9364.19 / 49.70
GTE65.12 / 50.0965.98 / 50.8566.85 / 52.0267.25 / 52.4968.00 / 53.36
Youtu63.58 / 48.8564.57 / 49.9466.50 / 52.0166.53 / 51.9167.80 / 53.51

With a shared teacher, full-document KD stays close to the starting score on KaLM (52.14 vs. 52.03 nDCG@10) and falls slightly on Jina (46.73 vs. 46.83). Using section scores improves on full-document KD by 1.17 to 2.07 points across the four backbones. Source-specific teachers already give stronger results than the shared teacher without any section scores. Adding section scores to those same teachers gives a further 0.77 to 1.60 points. The new RQ6 reports both controls in Section 4.7, Table 6, lines 491–520.

W2. Performance Degradation after Mixed-task SFT

“Furthermore, mixed-task SFT actually degrades performance for some backbones (Jina WixQA: 61.64->55.21; Youtu MIRACL: 73.30->69.86, with Youtu's average performance remaining nearly unchanged). Thus, some of the gains may reflect recovery from poorly tuned SFT rather than the proposed method itself.”

A: The two examples you cite do show recovery. On Jina/WixQA, nDCG@10 falls from 61.64 to 55.21 after Mixed-task SFT. StrataDistill with source-specific teachers brings it back to 61.63. On Youtu/MIRACL, the corresponding scores are 73.30, 69.86, and 73.50. To see whether recovery accounts for the broader pattern, we counted improvements and declines across the four backbones and five datasets in Table 2.

ComparisonRecall@10: up / downnDCG@10: up / downTotal: up / down
Mixed-task SFT vs. Original16 / 414 / 630 / 10
Task-specific SFT vs. Original19 / 116 / 435 / 5
StrataDistill with source-specific teachers vs. Mixed-task SFT20 / 020 / 040 / 0
StrataDistill with source-specific teachers vs. Original20 / 019 / 139 / 1

Compared with the original backbone, Mixed-task SFT lowers 10 of the 40 scores, by 1.59 points on average among those declines. Task-specific SFT lowers five scores, by 0.96 points on average. Mixed-task SFT already improves nDCG@10 in 14 of the 20 backbone and dataset combinations. The student trained with source-specific teachers improves on SFT in all 14 (lines 321–323).

The gains also hold against stronger starting points. Across backbones, the macro-average gains of this student are:

BaselineGain in Recall@10Gain in nDCG@10
Mixed-task SFT1.65 to 4.222.85 to 4.66
Task-specific SFT0.58 to 3.380.62 to 3.78
Original backbone5.98 to 8.194.68 to 7.68

Some individual gains are recoveries, but the gains in settings where SFT already helped cannot be explained that way.

AuthorsOpenReview ↗

(2/4) Response to Reviewer Uejm

W3. Mixing Ratio between Document and Section Scores

“The paper also does not investigate different mixing ratios between document-level and section-level scores.”

A: We varied the relative weights of the full-document and best-section scores with source-specific KaLM teachers. These are the five-task means from the weight sweep in Appendix C.1, Table 9, lines 900–911.

Document WeightSection WeightConfigurationAvg. Recall@10Avg. nDCG@10
0.001.00Section-only teacher68.8154.30
0.250.75Section-dominant teacher69.0954.85
0.500.50Balanced teacher (main setting)69.0854.86
0.750.25Document-dominant teacher68.9254.55
1.000.00Document-only teacher68.2253.72

The section-only target outperforms the document-only target on both metrics. Combining the scores works better still. Equal weights give the highest nDCG@10 in this sweep, while giving sections 75% of the weight gives the highest Recall@10. The two settings differ by only 0.01 on each metric, so the results support the equal-weight default without suggesting that exactly 50% is necessary.

W4. Native Sections versus Fixed-length Chunks

“Moreover, the contribution of structure awareness, as implied by the title, appears limited. Using the original section structure yields only a 0.29-point improvement in nDCG@10 over fixed-length chunking (54.88 vs. 54.59), while fixed-length chunks retain 89.8% of the gains. This suggests that the results support the use of maximum chunk-level similarity as a teacher score more strongly than they support structure awareness per se. Since the 0.29-point difference is based on a single run, its statistical significance is also unknown.”

A: The 89.8% figure is correct: it measures the share of the gain over Mixed-task SFT retained by the best fixed length. There are two comparisons here. The first asks whether local matches help at all. With the same source-specific KaLM teachers, full-document KD scores 53.72 nDCG@10, while adding 128-token chunks raises it to 54.59. The second asks how to choose the teacher views. Switching only the boundaries from those chunks to native sections raises the score to 54.88. The student still retrieves with one vector per document in all cases.

Local views for trainingRecall@10nDCG@10Gain over SFTShare of native-section gain
Mixed-task SFT66.7452.030.00—
StrataDistill / fixed views (128 tokens)68.8454.59+2.5689.82%
StrataDistill / fixed views (256 tokens)68.8054.53+2.5087.72%
StrataDistill / fixed views (512 tokens)68.7454.35+2.3281.40%
StrataDistill / fixed views (1,024 tokens)68.6254.26+2.2378.25%
StrataDistill / native sections69.0854.88+2.85100.00%

All four fixed lengths improve on SFT, by 2.23 to 2.56 nDCG@10 points. The best fixed length is 128 tokens on ESG, MultiDoc2Dial, and QASPER, 256 on MIRACL, and 1,024 on WixQA. Native sections give the best macro scores on both metrics and beat every fixed length in nDCG@10 on four of five datasets. MultiDoc2Dial is the exception. Even selecting the best length separately for each dataset leaves the fixed-chunk mean about 0.25 points below native sections. These results favor native boundaries when available without requiring them when the document has none. We have not tested the 0.29-point difference for significance. Section 4.4 and Figures 4–5 report these comparisons in lines 415–431.

AuthorsOpenReview ↗

(3/4) Response to Reviewer Uejm

W5 & Q2. Training–evaluation Corpus Overlap

W5: “Limitations of the evaluation setup: the same corpus, synthetic queries, and relatively short documents. All training queries are generated by Qwen3-Max from the very same document collections used for evaluation (e.g., 488 documents in MultiDoc2Dial and 1,585 in QASPER). Even the ESG test queries are synthetic. Consequently, the improvements may reflect adaptation to the specific corpus rather than better long-document retrieval in general. Deduplication using a cosine similarity threshold of 0.85 does not resolve this concern.”

Q2: “Training–evaluation corpus overlap and zero-shot generalization: Were the training queries generated from the same documents used in the test corpus?”

A: Yes. Qwen3-Max generated training queries from the same document collections used for evaluation. The cosine threshold of 0.85 filters near-duplicate queries, not shared documents. The table gives the percentage of documents relevant to test queries that also appeared as positive documents in training.

DatasetDocuments in the collectionTest-positive documents also used as training positives
ESG59,77125.85%
MIRACL100,00035.32%
MultiDoc2Dial48891.58%
QASPER1,58598.74%
WixQA6,22137.65%

The overlap is high on the two smallest collections, particularly QASPER. It is below 40% on ESG, MIRACL, and WixQA, and the source-specific student also improves over Mixed-task SFT on those three tasks. The five evaluations measure adaptation to new queries over a known collection. ESG test queries are synthetic as well. We state this setup in Appendix A.1, lines 763–767. W6 below reports a separate test of transfer to other collections.

W6. Evaluation on External Long-document Benchmarks

W6: “The method is not evaluated on established long-document retrieval benchmarks that are not used for training, such as LongEmbed, MLDR, LoCoV1, or MS MARCO Document.”

A: We trained the Mixed-task SFT baseline and both StrataDistill students on the five main long-document tasks: ESG, MIRACL, MultiDoc2Dial, QASPER, and WixQA. We tested those same checkpoints directly on LongEmbed and on LoCo excluding QASPER, without fine-tuning on either benchmark. Original is the released backbone. Both students improve over Mixed-task SFT for every backbone on both benchmarks. Parentheses give the nDCG@10 gain over Mixed-task SFT.

BenchmarkBackboneOriginalMixed-task SFTStrataDistill: shared teacherStrataDistill: source-specific teachers
LongEmbedKaLM55.2562.9464.31 (+1.37)63.77 (+0.83)
LongEmbedJina45.4755.4956.33 (+0.84)56.12 (+0.63)
LongEmbedGTE60.8667.5868.33 (+0.75)67.95 (+0.37)
LongEmbedYoutu54.7058.9160.74 (+1.83)61.35 (+2.44)
LoCo excl. QASPERKaLM72.2776.1577.42 (+1.27)77.44 (+1.29)
LoCo excl. QASPERJina63.0966.9667.34 (+0.38)67.55 (+0.59)
LoCo excl. QASPERGTE72.7678.0078.76 (+0.76)78.47 (+0.47)
LoCo excl. QASPERYoutu66.3869.8771.73 (+1.86)72.35 (+2.48)

On LongEmbed, the larger gain of the two configurations for each backbone ranges from 0.75 to 2.44 points. On LoCo excluding QASPER it ranges from 0.59 to 2.48. The shared teacher leads on three of the four LongEmbed backbones, while source-specific teachers lead on three of the four LoCo backbones. Section 4.2, Table 3, lines 324–345 gives the full results.

Comments on the paper

2 comments

Dear Reviewers,

Thank you again for your thoughtful feedback. We have posted our point-by-point responses in the individual review threads, where we also note the changes and additions to the manuscript.

We would welcome your thoughts on our responses and would be happy to discuss any remaining questions before the discussion period ends.

Best regards, The Authors

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank Reviewers Uejm, vsDq, and DbTJ for their thoughtful comments, and the Area Chair for coordinating the discussion. We summarize the contribution of StrataDistill and the evidence addressing the main concerns below.

StrataDistill uses document structure as supervision for learning single-vector representations. It is an adaptive training method that requires neither a stronger teacher nor section-level annotations and leaves inference unchanged. The same training rule supports short documents without decomposition. Experiments show gains across four embedding backbones and transfer to external long-document benchmarks. In joint training, KaLM retains short-document performance while improving long-document retrieval.


1. Structure contributes beyond KL distillation and expert consolidation

To address the shared concern from Uejm, vsDq, and DbTJ, RQ6, Table 6 holds the teacher and training setup fixed and changes only the scoring rule. Adding section scores improves mean nDCG@10 on all four backbones with both shared and source-specific teachers. These controls establish an additional contribution from structural evidence beyond KL self-distillation and expert consolidation.

2. Improvements transfer beyond the training collections

In response to Uejm and vsDq, we directly evaluated checkpoints trained on the five main tasks on LongEmbed and LoCo excluding QASPER, without further tuning. Both StrataDistill variants outperformed mixed-task SFT on every backbone and both benchmarks in Table 3. Appendix A.1 clarifies the training data and document lengths, including the related question from DbTJ.

3. Repeated runs and paired comparisons support reliability

Both Uejm and vsDq raised concerns about single-run results. Across three seeds, mean nDCG@10 for KaLM with source-specific teachers spanned 0.02 points. For the source-specific student versus mixed-task SFT, all 20 dataset-level 95% paired bootstrap confidence intervals for the gain were above zero. Appendix C.4 reports these analyses.

4. Comparison with FGD

In response to vsDq, we clarified that the distinction from FGD lies in how local evidence becomes supervision. FGD, cited in the original submission, augments document-level contrastive learning with passage- and sentence-level supervision from an external cross-encoder. StrataDistill instead combines full-document and best-section scores from same-backbone teachers into document-level ranking targets. The student learns these rankings using only full-document scores. Our contribution is to turn structural evidence into supervision for whole-document retrieval, rather than separately distill relevance at multiple granularities.

We could not obtain the FGD training code, so we also implemented the published method. After verifying a gain over a matched control on an MS MARCO subset, we adapted it to long documents. From the same KaLM checkpoint, StrataDistill with a shared teacher reached 53.64 mean nDCG@10, compared with 51.93 for our best FGD adaptation.

Our response to DbTJ also acknowledges that the 8B embedding teacher performs better on QASPER. RQ6 separately measures the benefit of section scores with the teacher held fixed.


In follow-up, Uejm confirmed that the additional experiments addressed the main concern and recognized the evidence on generalization and reliability. The reviewer raised the rating to Weak acceptance. We appreciate this follow-up and would have welcomed further discussion in the remaining review threads.

We believe our responses and additional experiments address the main concerns raised in the reviews. We would be grateful if you could consider our rebuttal, revised manuscript, and the latest discussion in your final assessment.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.