acceptodds
Under review as a conference paper at ICLR 2027

DRIFT: Breaking Domain Entanglement in Multi-Source Out-of-Distribution In-Context Learning

Abstract

In this paper, we study multi-source out-of-distribution in-context learning (OOD ICL), where demonstrations from heterogeneous source domains are used to support inference on an unseen target distribution for Large Language Models (LLMs). We reveal that increasing source diversity can unexpectedly degrade ICL performance. To explain this phenomenon, we establish a Bayesian framework that decouples demonstration retrieval from LLM inference, and theoretically characterize how domain information entangled in retrieval representations propagates source-dependent bias into the predictive distribution. Our analysis further shows that suppressing such domain information yields a tighter bound on the divergence from the target predictive distribution. Motivated by this result, we propose a domain-disentangled retrieval framework that preserves transferable semantics while minimizing domain-specific information during demonstration selection. Experiments on multiple LLMs and OOD benchmarks consistently validate our theoretical findings and demonstrate improved robustness under multi-source shifts, establishing a new perspective for multi-source OOD ICL.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
In-Context Learning, Out-of-Distribution Generalization, Multi-Domain Learning, Demonstration Retrieval, Domain Disentanglement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jM2pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies in-context learning when the demonstrations come from multiple source domains and the target domain is unseen. The authors first show that retrieving from the merged pool can work worse than retrieving from a single well-chosen source domain. They then give a Bayesian argument: residual domain information in the retrieval representation brings a source-dependent bias into the LLM's prediction, so removing this information tightens a divergence upper bound. Based on this, the authors propose DRIFT, a small retriever trained with a domain-adversarial branch, an orthogonality constraint, a class-conditioned cross-domain contrastive loss, and prototype-guided scoring. Only the "semantic" branch is used at retrieval time. Experiments cover SA, TD, NLI and NER, 16 datasets in total, with five local LLMs and two API LLMs. The comparison is against kNN, DICL, UCS and CCL. There is also an ablation of the four components, and an analysis of the retrieval subspaces and of source-domain heterogeneity.

Critical strengths
  1. The problem is real. Pooling demonstrations from multiple source domains is something people do in practice. Among the work I have seen, this paper is the first to ask whether doing so helps or hurts in ICL. The method is tested on four tasks, 16 datasets and seven LLMs. It beats the strongest baseline in 18 of 20 model-task combinations (Table 1). I recomputed the averages in Table 1 and Table 2 from the per-dataset tables in the appendix, and they match.
  2. The analysis goes beyond the main table. There is a per-component ablation (Table 2). There are results for retrieval with the semantic branch, the domain branch, and both together (Fig. 4). The paper also looks at how the gain changes with source-domain heterogeneity (Fig. 5). Results are reported for every dataset, including the cases where DRIFT loses.
Critical weaknesses
  1. The method is complex, but the paper does not show that this complexity is needed. DRIFT stacks five losses (classification, domain, adversarial, orthogonality, contrastive), plus prototype scoring. Removing any one component costs at most 1.16 points on Llama3.2-1B and at most 2.47 points on Qwen3-1.7B (Table 2). The full method is 10.83 and 3.54 points above plain kNN on these two models (Table 1). So most of the gain cannot be explained by any of the components the paper discusses. Fig. 4 points the same way. Retrieving with the domain branch z_d, which the method is meant to discard, still beats all four baselines on Qwen3-1.7B. On Llama it also beats kNN, DICL and UCS. The simplest explanation is that a supervised retriever helps by itself, not that removing domain information helps. The ablation is also run only on the two smallest models (1B and 1.7B). On the strong API models (Qwen3.5-Plus, DeepSeek-V4-Flash) the comparison is only against CCL, with 200 samples per domain, and the margin shrinks to 3.89 and 1.77 points. It is not clear whether these components still matter once the base model gets stronger. The control I need to see is a retriever trained with only the classification loss, run on both a small LLM and a strong LLM. If the authors can show that the full design is necessary, I would consider raising my score.
  2. Part of the evaluation does not measure retrieval quality. For NLI, the premise and hypothesis are concatenated first and then truncated to 100 tokens (dataloader.py). On ContractNLI the median premise length is about 1,400 words, so the hypothesis is cut off in every test sample. The largest effect in Fig. 1 (−17.1 on Qwen ContractNLI) and the main ablation effect on NLI (Table 10) both come from this dataset. The paper reports only accuracy. On CivilComments (92% benign), all methods, including fine-tuning, fall below majority-class accuracy.
  3. Related work. The OOD generalization part of Sec. 2 cites nothing newer than 2022, and the methods it cites are all from 2016-2021. Please add recent work on separating semantic and domain information, for example ProS (Fang et al., CVPR 2024), Disentangled Prompt Representation for Domain Generalization (Cheng et al., CVPR 2024) and DePro (Chen et al., SIGIR 2025).
Questions for authors
  1. How much does a retriever trained with only the classification loss (same backbone, same prototype scoring) achieve? Please report it on at least one small LLM and one strong LLM. If it comes close to DRIFT, the paper should say so.

  2. Please run the component ablation on Qwen3.5-Plus or DeepSeek-V4-Flash. Do the components still change the results?

  3. After truncation on ContractNLI, can the model still see the hypothesis? Please report macro-F1 in all tables and add a majority-class row.

2 comments

Deeeeeeeeeear reviewer, has my response cleared up your concerns? Please drop by and check out my reply when you have time🙏

I thank the authors for the additional experiments. The response resolves my main question in part. A retriever trained with only the classification loss falls 7.11 points below DRIFT on Llama3.2-1B and 3.32 points below it on Llama3.1-8B, so supervised retriever training alone does not explain the gain. This is the control I asked for, and it supports the method as a whole. Two concerns remain. First, the response does not show that each component is needed. On Llama3.2-1B, removing any single component costs at most 1.16 points in Table 2, while removing all the added training objectives costs 7.11, so the components appear largely interchangeable and a simpler variant may do as well. On the strong models the new ablation covers only sentiment analysis with 200 samples per dataset. On Qwen3.5-Plus the full method is ahead of each ablated variant by 0.25 to 0.75 points, which is 2 to 6 of 800 samples, and it is the best variant on one of the four datasets. Second, the corrected ContractNLI results weaken the NLI evidence. DRIFT is now ahead of kNN on one of the four models. If I replace only the ContractNLI column in the original per-dataset table, the NLI gain over kNN on Qwen3-1.7B falls from 3.02 to 0.02 points. The response does not rerun Fig. 1 or the NLI ablation, whose largest effects came from this dataset, and the macro-F1 table compares DRIFT only with CCL. I also could not find a revised PDF, so I cannot check how these results enter the paper. I keep my rating at 2.

Reviewer_NwSPOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper demonstrates the necessity of purely semantic retrieval by deriving and comparing the upper bounds on Bayesian predictive divergence between retrieval with and without domain bias. As its methodology, the authors employ domain-adversarial learning and representation orthogonality regularization to disentangle semantic and domain information in the learned representations. Subsequently, to achieve domain-invariant alignment, they apply contrastive learning to bring examples belonging to the same class closer together in the semantic space and propose a prototype-based demonstration selection method. The authors train a single retriever and evaluate it across seven LLMs, demonstrating superior performance compared to the baselines.

Critical strengths

The paper is very well structured, with a clear logical flow that makes it easy to read and follow. Given how difficult it has become to read many recent AI-generated academic texts, it was refreshing to encounter such a well-organized paper. Since the retriever does not rely on LLM feedback, the authors demonstrate its applicability across multiple LLMs using a single retriever, highlighting the scalability and generalizability of the proposed method. The paper also includes appropriate ablation studies and analytical experiments.

Critical weaknesses

Minor 1. Class-conditioned Cross-domain Alignment, one of the proposed methodological components, appears to be applicable only when the demonstration pools from multiple domains share the same set of class labels. How can this approach be extended to multi-domain demonstrations with different class labels? Although the paper focuses on OOD scenarios within a single task, where the class labels are assumed to remain consistent across domains, its applicability to multi-task ICL appears to be limited.

Major 1. The authors interpret ICL as Bayesian inference and build their theoretical analysis upon this formulation. However, a fundamental question arises: Can ICL be adequately modeled as Bayesian inference? In ICL, the types, ordering, and number of demonstrations can all significantly affect the final output. However, the formulation in Definition 1 appears to require the assumption that demonstrations contribute independently. In particular, the product-based formulation does not account for ordering effects. Given the gap between actual LLM inference and the proposed mathematical modeling, the authors should provide sufficient justification for this assumption and discuss its limitations.

Major 2. Section 3 theoretically derives the difference in the upper bounds on Bayesian predictive divergence with and without domain bias. However, the derived upper bounds are not directly utilized in the design or optimization of the proposed method, and it remains unclear whether this analysis provides any new insights beyond the observation that domain bias can affect retrieval. Therefore, the practical contribution of this theoretical analysis appears to be limited.

Major 3. In in-context learning, semantic relevance alone may not be sufficient as a criterion for demonstration selection. While using an independent retriever without relying on LLM feedback is an advantage, the proposed approach may fail to adequately capture the actual context utility for LLMs. In particular, since the method selects the Top-K demonstrations based on similarity, it does not consider set-level properties of the demonstration set, such as diversity and ordering. Further analysis is needed to investigate how these factors affect OOD ICL performance.

Questions for authors

The questions are summarized below. Please refer to the Weaknesses section for detailed explanations.

Minor 1: How can Class-conditioned Cross-domain Alignment be extended to multi-task ICL with different class labels?

Major 1: How can the Bayesian formulation of ICL justify the independence assumption and account for demonstration ordering effects?

Major 2: What additional insights or practical benefits does the theoretical upper-bound analysis provide?

Major 3: How does the proposed method account for LLM context utility and set-level properties such as demonstration diversity and ordering?

Reviewer_wNP8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DRIFT, a demonstration retrieval framework for multi-source out-of-distribution (OOD) in-context learning. The method addresses domain entanglement by separating task-relevant semantic representations from domain-specific information through adversarial learning, orthogonality regularization, and class-conditioned cross-domain alignment. It further introduces prototype-guided retrieval to select demonstrations based on both instance-level and class-level semantic similarity. The paper also provides a Bayesian analysis of how domain entanglement affects retrieval quality and predictive divergence. Experiments across four NLP tasks, 16 datasets, and seven LLMs demonstrate consistent improvements over existing retrieval baselines, supported by ablation studies and analyses of representation disentanglement and source-domain heterogeneity.

Critical strengths

S1: The paper provides a meaningful theoretical analysis of domain entanglement in multi-source OOD in-context learning. Specifically, it introduces a Bayesian framework that connects semantic retrieval cost to predictive divergence and characterizes how multi-source domain heterogeneity introduces additional terms into the theoretical upper bounds, providing motivation for disentangled retrieval.

S2: The proposed method effectively combines domain disentanglement with class-conditioned cross-domain alignment. Specifically, it separates task-relevant semantic representations from domain-specific information and aligns same-class examples across domains. The ablation studies and representation analyses provide empirical evidence supporting the contributions of these components to improved OOD retrieval performance.

Critical weaknesses

W1: The experimental comparison is limited in terms of retriever backbones and controlled baseline evaluation. Specifically, DRIFT relies on all-MiniLM-L6-v2 without evaluating stronger embedding models, leaving its effectiveness with more capable retrievers unclear. Furthermore, although CCL [1] also learns disentangled representations, the comparison does not apply the same prototype-guided retrieval strategy to CCL, making it difficult to isolate the contribution of DRIFT’s representation learning.

W2: The evaluation covers a limited range of task types, restricting the evidence for broader OOD generalization. Specifically, the experiments focus on sentiment analysis, toxicity detection, natural language inference, and named entity recognition, without evaluating reasoning-oriented tasks such as commonsense or mathematical reasoning, which have been considered in prior OOD ICL research [1]. Consequently, it remains unclear whether the proposed approach generalizes beyond the evaluated task families.

References [1] CCL: Causal-aware In-context Learning for Out-of-Distribution Generalization. NeurIPS 2025.

Questions for authors

Q1: Fidelity and Fairness of Baseline Comparison
Was CCL reproduced using its original training objectives and implementation? Were all retrieval baselines evaluated under comparable training and hyperparameter tuning conditions? In particular, how would CCL perform if equipped with the same prototype-guided retrieval strategy as DRIFT?

Q2: Evidence for Improved Disentanglement
Does DRIFT achieve better semantic-domain disentanglement than CCL under the same evaluation setting? Can the authors provide direct evidence that the performance improvements stem from the proposed disentanglement and class-conditioned alignment mechanisms, rather than differences in retrieval strategies?

2 comments

Deeeeeeeeeear reviewer, I think my response has addressed your concerns. I would be incredibly grateful if you could take a moment to review my reply🙏🙏🙏

Thank you for your response. After reviewing it, I am maintaining my original rating. I remain concerned that the proposed encoder-based method offers limited novelty relative to the encoder–decoder-based CCL approach and that the experimental analysis lacks sufficient depth and direct evidence of representation separation.

Comments on the paper

1 comment

Dear reviewers,

As the discussion period is limited, please follow up on the latest responses in a timely manner to address any remaining concerns. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.