acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Structured Transformation: Mitigating Distribution Shift in Dense Retrieval Through Training-Time Preprocessing

Abstract

Dense retrieval models are trained assuming that finetuning on task-relevant queries improves performance, yet this assumption can break down when training data contains synthetic components or originates from a misaligned distribution with target tasks. We find that in such scenarios, naively finetuning on seemingly relevant data can result in negative transfer, causing significant degradations over not finetuning at all. We propose Adaptive Structured Transformation (AStrucT), an automatic preprocessing technique that leverages off-the-shelf Large Language Models (LLMs) to organize training documents into domain-specific structures prior to finetuning. These domain-specific schemas are induced automatically from a small sample of target-domain passages, without manual schema design or access to test-time queries. Across three model scales and twelve diverse domains (BRIGHT), AStrucT yields an average improvement of 3.75 percentage points (pp) nDCG@10 over direct finetuning, and 1.25 pp over the pretrained baseline, consistently mitigating negative transfer. For the 4B model, domain-matched templates outperform general rewriting and templates from other domains on average. These findings provide a practical strategy for adapting embedding models using automatically prepared training documents, without requiring LLM calls at inference.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dense retrieval, distribution shift, negative transfer, document expansion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZiprOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors presented Adaptive Structured Transformation (AStrucT) where they transform raw documents into domain specific structure document that follows a learned schema. Essentially they are rewriting document from the view point of an expert in a domain using an LLM. The authors argue that this can eliminate misaligned distribution in original documents with the target task, and that transformed documents are better source for finetuning. On BRIGHT dataset, they show that fine tuning on original documents reduced performance compared to no fine tuning whereas using transformed documents improved performance.

Critical strengths
  1. It consistently improved performance over baseline and direct finetuning. The gains are consistent across three model scales (0.6B, 4B, 8B)
  2. Mixing as well as prepending original + transformed document, both are worse than just using transformed data. This presents a stronger case for using transformed documents.
  3. The approach is simple and has no inference-time cost
Critical weaknesses

None to support rejection

Questions for authors
  1. How much of this is because we used a stronger model to rewrite?
  2. How can we confirm that the problem of HQ FT is that the synthetic data is misaligned?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Zipr

Thank you for the supportive review and for highlighting the mixing and prepending ablations.

Q1: Rewriter strength. We don’t believe rewriter strength is as important as how the LLM is instructed to rewrite. GPT-4.1 is used for all our experiments, both generic rewriting (only +0.19 over HQ) and AStruct (+3.96 over HQ).

Q2: Misalignment. Our empirical results establish negative transfer and suggest misalignment, and even our embedding analysis is consistent with the explanation that HQ data does not match the test distribution. To dig deeper though, we can also look at where the documents come from. ReasonIR-HQ uses BRIGHT documents as positives, generates queries from those documents, and then generates hard negatives conditioned on the query and positive document. Their generation prompt does not instruct the model to align the document to match style or distribution with the real BRIGHT queries or documents, however it does encourage matching domain. This further suggests that dataset misalignment is the main problem.

Reviewer_g6chOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AStrucT, a training-time document transformation method for mitigating negative transfer in dense retrieval. It uses an LLM to generate domain-specific schemas and transform training documents while preserving queries and relevance assignments. Experiments on twelve BRIGHT domains using three Qwen3-Embedding model sizes show improvements over direct fine-tuning and pretrained baselines.

Critical strengths
  1. Meaningful practical problem.

The paper identifies substantial negative transfer when dense retrievers are fine-tuned on synthetic query-document training data. The proposed training-time preprocessing approach is simple and requires no additional inference-time computation.

  1. Useful empirical evaluation.

AStrucT improves average nDCG@10 across three model sizes. Comparisons with general rewriting, wrong-domain templates, and data-mixing strategies provide useful evidence supporting domain-specific transformations.

Critical weaknesses
  1. Missing statistical significance tests for retrieval effectiveness.

The paper reports extensive nDCG@10 comparisons in Tables 1–4 but provides no paired significance tests, p-values, or confidence intervals for the main retrieval improvements. This is particularly important because the average gains over the pretrained models are relatively modest (1.02–1.61 nDCG@10 points), and individual domains contain only 76–194 evaluation queries.

Without query-level statistical testing, the reliability of the reported improvements over No FT, HQ FT, and alternative transformation methods remains unclear.

  1. Limited generalizability and incomplete analysis of negative-transfer mitigation.

The experiments are restricted to the BRIGHT benchmark, ReasonIR-HQ training data, and the Qwen3-Embedding model family. It is therefore unclear whether the proposed method generalizes to other retriever architectures or types of training-distribution mismatch.

Furthermore, the observed negative transfer may partly depend on the fine-tuning configuration. The paper does not sufficiently compare against more conservative fine-tuning strategies, such as smaller learning rates or stronger regularization. Consequently, it remains unclear whether structured transformation is necessary to mitigate the observed degradation.

Questions for authors
  1. Can the authors report query-level paired significance tests (e.g., paired t-tests or randomization tests) for AStrucT versus No FT, HQ FT, and the main rewriting baselines? How are multiple comparisons handled?

  2. Have the authors evaluated whether more conservative fine-tuning configurations reduce the observed negative transfer?

  3. Is there evidence that AStrucT generalizes beyond the ReasonIR-HQ/BRIGHT setup or the Qwen3-Embedding family?

2 comments
AuthorsOpenReview ↗

New Statistical Analyses and E5-Mistral Experiments Strengthen the Evidence for AStrucT

Thank you for the statistical and tuning suggestions!

W1: Significance tests (Q1). Our improvements are significant: we now report paired query-level 95% confidence intervals for the original Table 1 results using 20,000 bootstrap resamples, preserving equal evaluation weights and shared-query dependencies. All six individual, unadjusted intervals exclude zero, supporting AStrucT’s positive average gains over both HQ FT and No FT at all three Qwen3 scales. For the new E5 experiments, we additionally report two-sided paired sign-flip tests with Holm correction within each family of related comparisons. At 6e-6, AStrucT’s twelve-domain E5 gains over both HQ and No FT remain significant after correction (Holm p<.001). The confidence intervals and corrected E5 results are provided in our general response.

W2: Generality and conservative fine-tuning (Q2, Q3). On E5-Mistral-7B, we compared two checkpoint-selection strategies: selecting the checkpoint with the lowest development loss and using a fixed checkpoint at step 50. Across the two configurations reported below, AStrucT achieves higher mean nDCG@10 than HQ.

On four BRIGHT domains using ReasonEmbed training data (as suggested by Reviewer Kiw9), AStrucT scores 32.07 versus 34.62 for ReasonEmbed-document training. The specific set-up is described in our reply to Reviewer Kiw9.

Interestingly, this suggests that conservative fine-tuning might actually exacerbate negative transfer.

CheckpointNo FTHQAStrucTAS − HQ95% query CIHolm p
Lowest dev loss14.8815.3618.06+2.70[1.76, 3.65]<.001
Step 5014.8813.9916.84+2.86[1.85, 3.89]<.001

Our Qwen3 experiments demonstrate negative transfer and its mitigation in the tested setup. The E5 extension evaluates whether AStrucT’s advantage over direct HQ fine-tuning extends to another retriever; it does not assume that HQ fine-tuning harms every model.

Hi! Thank you again for your helpful feedback. Our rebuttal reports new E5-Mistral-7B results across all twelve BRIGHT domains, extending the evaluation beyond Qwen3. We would love your thoughts before the response period ends tomorrow EOD!

Reviewer_Kiw9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Adaptive Structured Transformation (AStrucT), a training-time document preprocessing method for mitigating negative transfer in dense retrieval. An LLM first generates a domain-specific schema and transformation guidelines from five example passages. The resulting template is then used to transform positive and negative training documents into structured text, while preserving the original queries and relevance assignments. The retriever is fine-tuned on these transformed documents and retrieves original documents at inference. Experiments use ReasonIR-HQ training data and twelve BRIGHT evaluations with three Qwen3-Embedding model sizes. AStrucT improves average nDCG@10 by 3.75 percentage points over direct fine-tuning and 1.25 points over the pretrained models, averaged across the three sizes.

Critical strengths
  1. The problem of negative transfer from synthetic training data is practical. AStrucT provides a simple preprocessing method without additional LLM calls at inference.

  2. AStrucT improves average retrieval performance over direct fine-tuning at all three model sizes. The rewriting and data-mixing ablations provide useful analysis of the method.

Critical weaknesses
  1. The practical value of AStrucT is insufficiently demonstrated. ReasonEmbed [1] reports 38.1 nDCG@10 on BRIGHT using original queries, and its 8B variant trained with basic InfoNCE already achieves 37.1, compared with 22.03 for AStrucT-8B. Although the training recipes differ, evaluating only on ReasonIR-HQ leaves it unclear whether AStrucT benefits stronger synthetic supervision. Applying it to the released ReasonEmbed training data with a fixed initialization and training procedure would better demonstrate its potential to improve a competitive retriever.

  2. The rewriting controls do not isolate the benefit of domain-specific schema induction. General rewriting preserves the source content and approximately its length, while AStrucT extracts summaries and concepts and encourages conciseness. Wrong-domain transformation changes the expert instruction, schema, and guidelines together. A comparison with a fixed, domain-independent summary-and-key-concepts template at comparable output lengths is needed to establish whether generating a separate schema for each domain is necessary.

[1] ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval: https://aclanthology.org/2026.acl-long.54/

Questions for authors
  1. Could you evaluate AStrucT on the released ReasonEmbed training data [1], comparing original and transformed training documents with the same initialization, loss function, query-document assignments, and training budget, and include ReasonEmbed as a reference baseline? This would establish whether AStrucT complements stronger synthetic supervision and can further improve an already competitive retrieval model.

  2. Could you compare AStrucT with a fixed, domain-independent template containing a summary and key concepts, using the same LLM and comparable output lengths? This experiment would clarify whether inducing a separate schema for each domain is necessary for the reported gains.

  3. How stable are the gains over No FT across training seeds and different samples of five passages for template generation? Confidence intervals for the paired retrieval differences would help assess the relatively small average improvements over the pretrained models.

[1] ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval: https://aclanthology.org/2026.acl-long.54/

2 comments
AuthorsOpenReview ↗

AStrucT Mitigates HQ Negative Transfer while Preserving Existing Supervision

Thank you for taking the time reviewing our paper and the suggestions!

W1: Practical value and stronger supervision (Q1). Our contribution is mitigating negative transfer while retaining existing queries and relevance assignments. Our main intention is not a new supervision-generation pipeline or a state-of-the-art retriever. BRIGHT and the synthetic HQ dataset provide a controlled testbed for this adaptation question.

Nevertheless, thank you for bringing our attention to ReasonEmbed. Considering ReasonEmbed data was created specifically for BRIGHT, we do not expect to see negative transfer, and therefore cannot use it to effectively test whether AStrucT recovers. Still, we were also curious and completed a limited comparison on four BRIGHT domains: Biology, AoPS, StackOverflow, and Economics. We use the same MS MARCO-fine-tuned Qwen3-8B initialization, InfoNCE loss, labels, and training steps (438). Original-document training scores 34.62, versus 14.92 for noFT. Meanwhile, we get 32.07 for AStrucT (difference −2.55; paired 95% CI [−3.93, −1.20]); the separately released Basic reference scores 33.93. This suggests that AStrucT may not benefit when the training data is aligned, which does not dispute our findings, as we do not claim complementarity in this setting. Still, the work itself is quite similar, and we will update the related works section around synthetic data for reasoning, including ReasonEmbed, in our next version.

W2: Necessity of per-domain schemas (Q2). With GPT-4.1 used for all rewrites at 4B, AStrucT scores 21.97, compared with 18.20 for general rewriting and 20.00 for wrong-domain templates (Table 2). However, due to time and cost limitations, we have not tested whether a fixed summary-and-key-concepts template or whether producing text of similar length could achieve the same gains. We agree that a fixed, length-matched summary-and-key-concepts template would provide a more direct comparison. We therefore limit our claim to the benefit of domain-guided rewriting, without asserting that a separate schema is necessary for each domain. Note that, as we found in the length comparison, AStrucT-rewrites on average are longer than wrong-domain rewrites, suggesting that domain guidance may preserve critical information.

Q3: Stability. On twelve BRIGHT domains, E5-Mistral-7B trained at the paper’s 6e-6 learning rate improves over No FT by 3.18 nDCG@10 points with AStrucT (95% query-level CI [2.04, 4.36], Holm p<.001). This interval reflects query-level uncertainty for the fitted models under one training seed. We have not measured variability across training seeds or different five-passage samples due to cost; we prioritized breadth to target more datasets.

Hi! Thank you again for your helpful suggestions. Following your recommendation, our rebuttal includes a four-domain comparison using ReasonEmbed training data. Since the response period ends EOD tomorrow, we would appreciate your thoughts before tomorrow’s deadline!

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback! UtLm, Kiw9, and g6ch recognize negative transfer from synthetic training data as a useful and practical problem, and Kiw9, g6ch, and Zipr highlight AStrucT's simplicity. All four reviewers note that AStrucT adds no inference-time cost and improves average retrieval performance across model sizes, and all four find our controls and ablations informative. UtLm calls our work "a useful empirical contribution", and Zipr comments that there is no critical weakness to support rejection.

We address a few main concerns raised by multiple reviewers. First, we validate our findings with an additional embedding model, E5-Mistral-7B, and we find more evidence of the benefits of AStrucT. We also provide more insight into why AStrucT leads to gain and provide confidence intervals for our main results.

Beyond Qwen3 (UtLm, Kiw9, g6ch). We fine-tune E5-Mistral-7B on our paper’s HQ and AStrucT training documents with 6e-6 (our paper’s learning rate) and evaluated it on all twelve BRIGHT domains. Each run uses the checkpoint with the lowest development loss. The table reports paired nDCG@10 differences with 95% confidence intervals and Holm-adjusted p-values (across query sampling).

ComparisonDifference95% CIHolm p
AStrucT − HQ+2.70[1.76, 3.65]<.001
AStrucT − No FT+3.18[2.04, 4.36]<.001

While HQ fine-tuning does not lower the average performance in this primary E5 comparison, it still does in some domains (examples: economics: 12.9 NoFT to 12.2 HQ; earth science: 22.8 NoFT to 14.7 FT). Meanwhile, AStrucT improves on it on average. The extension validates AStrucT’s benefit beyond what we found for using Qwen3 models. Furthermore, these gains were obtained using the same learning rate as our Qwen3 experiments. Together, we believe this addresses g6ch W2/Q3.

AStrucT significantly outperforms both baselines, and its lead over HQ is similar even when every run is evaluated at the same early checkpoint (step 50: +2.86 points, Holm p<.001). We will include the complete per-domain breakdown in our next version.

Explaining the gains (Reviewer UtLm, Kiw9, Zipr). In the paper (Sec. 5.1), we explored some hypotheses for what AStrucT contributes. Two hypotheses included 1) data augmentation via rewriting; and 2) restructuring content into a templatic format. However, we found in these ablations that AStrucT scores 21.97, compared with 18.20 for a general rewriting transformation and 20.00 using a structured transformation template for the wrong domain. AStrucT therefore outperforms both tested rewriting alternatives with the same LLM. While we suspect that alternate LLMs will result in generally better representations, the primary purpose of our work is to study the embedding models and finetuning process, and so we considered the LLM fixed in our study. Furthermore, alternate LLMs to regenerate all of the artifacts would be costly.

UtLm W1 and Kiw9 W2 raise document length as a potential confound. The lengths of our rewrites are mostly comparable relative to the HQ document lengths (AStrucT: 165.7%; wrong-domain AStrucT: 140.7%; HQ: 100%; general rewriting: 89.6%) in terms of tokens. As HQ is the worst-performing on average and longer than general rewriting, we don’t believe length necessarily correlates. Additionally, it was out-of-scope to find exact length-matched, domain-agnostic instructions because domain guidance will result in more details retained in the rewrite. We further find that relatively few (<0.25%) of all passages are truncated at 2048 tokens, suggesting that truncation has little effect.

We will clarify the intent of section 5.1 and how it relates to structure and organization, as well as provide answers to the reviewer’s questions in the revision.

Confidence Interval for Table 1 (UtLm, g6ch). We computed individual, unadjusted 95% confidence intervals using 20,000 paired query-level bootstrap resamples of the original predictions, retaining equal weighting across the twelve evaluations and jointly resampling the 76 shared TheoremQA queries. All six intervals exclude zero, supporting positive average gains over both HQ FT and No FT at all three model scales. These results strengthen the empirical evidence that AStrucT effectively mitigates negative transfer from direct fine-tuning while improving average retrieval beyond the pretrained baseline in the evaluated settings.

Qwen3 sizeAStrucT − HQ FTAStrucT − No FT
0.6B+3.05 [2.20, 3.90]+1.62 [0.75, 2.50]
4B+3.97 [3.09, 4.85]+1.03 [0.23, 1.85]
8B+4.22 [3.42, 5.03]+1.12 [0.30, 1.93]

Planned revisions. We will add the new E5 results and length/truncation measurements, include the per-domain templates and CLERC example, and document the training configuration and checkpoint-selection procedure.

AuthorsOpenReview ↗

New Confidence Intervals Numbers and E5-Mistral Results Support AStrucT’s Gains over No Fine-Tuning

Thank you for the careful review and concrete suggestions.

W1: Isolating adaptive structure (Q1). We agree that it is important to properly isolate the effects of domain-specific instructions, and as we discussed in the general response, we will be clearer and adopt your narrower claim in our next version. In particular, we can more precisely clarify that Table 2 represents a domain-matched transformation over two alternatives, not structure in isolation, as domain instructions, content selection, length, and organization are being changed together. We tried our best to primarily isolate structure. We will revise Section 5.1 and the conclusion accordingly.

Still, we took a closer look at length to rule out some of these other factors. W1 suggests that AStrucT would result in shorter snippets and therefore summaries would be mismatched. We find that AStrucT is longer: domain-median rewritten/source length ratios range from 1.2 -- 3.5x, with the mean being 1.65x. We also observe similarly in wrong-domain: those outputs are long (1.4x), yet perform worse than AStrucT (where AStrucT being 17.7% longer). Note that domain-specific rewrites will encourage more information retention and therefore slightly longer texts on average: length and domain-specificity for structured transforms are tied.

Finally, 0.24% of rewritten passages and 0.23% of HQ passages are truncated at 2,048 tokens (see general response, “Explaining the gains”). These measurements compare AStrucT with raw HQ documents. They are not meant to establish any major conclusions around length between the three rewriting methods, besides ensuring it is controlled well-enough (that we don’t have 10x or 0.1x documents). We will report these statistics with this scope made explicit.

W2: Uncertainty relative to No FT (Q2). Using the original Table 1 predictions, we computed individual, unadjusted 95% confidence intervals from 20,000 paired query-level bootstrap resamples. We retained equal weighting across the twelve BRIGHT evaluations and jointly resampled the 76 shared TheoremQA queries. AStrucT’s macro-average gains over No FT are +1.62 [0.75, 2.50], +1.03 [0.23, 1.85], and +1.12 [0.30, 1.93] nDCG@10 percentage points for Qwen3 0.6B, 4B, and 8B, respectively. All three intervals exclude zero.

Our new E5-Mistral-7B experiment provides complementary evidence, with a gain over No FT of +3.18 points (95% CI [2.04, 4.36], Holm-adjusted p<.001; see General Response).

For schema induction, we randomly selected five passages per domain, generated each domain schema once, and kept it fixed for reuse. These intervals measure query-level uncertainty for the fitted models; we have not measured variability across training seeds or different five-passage samples due to cost constraints; we prioritized for breadth (average effects across datasets).

Q3: Checkpoint selection and reproducibility. Checkpoints are selected by the lowest development contrastive loss. We use the 10% ReasonIR-HQ development split, with the training objective and sampled positives, hard negatives, and in-batch documents. No BRIGHT evaluation query is used for checkpoint selection. Training uses a learning rate of 6e-6, a warmup ratio of 0.06, a 2,048-token limit, and at most 700 steps, with evaluation every 10 steps and early-stopping patience of five evaluations. The effective batch size is 192; 8B uses batches of 96 with two accumulation steps and thus a smaller in-batch negative pool. We will document the training configurations and include the actual per-domain templates and instantiated CLERC demonstration into the appendix.

Hi! Thank you again for your careful review! Our rebuttal now includes paired query-level confidence intervals for the Table 1 comparisons with No FT. With the response period ending EOD tomorrow, we would appreciate your thoughts on our response!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.