- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies in-context learning when the demonstrations come from multiple source domains and the target domain is unseen. The authors first show that retrieving from the merged pool can work worse than retrieving from a single well-chosen source domain. They then give a Bayesian argument: residual domain information in the retrieval representation brings a source-dependent bias into the LLM's prediction, so removing this information tightens a divergence upper bound. Based on this, the authors propose DRIFT, a small retriever trained with a domain-adversarial branch, an orthogonality constraint, a class-conditioned cross-domain contrastive loss, and prototype-guided scoring. Only the "semantic" branch is used at retrieval time. Experiments cover SA, TD, NLI and NER, 16 datasets in total, with five local LLMs and two API LLMs. The comparison is against kNN, DICL, UCS and CCL. There is also an ablation of the four components, and an analysis of the retrieval subspaces and of source-domain heterogeneity.
Critical strengths
- The problem is real. Pooling demonstrations from multiple source domains is something people do in practice. Among the work I have seen, this paper is the first to ask whether doing so helps or hurts in ICL. The method is tested on four tasks, 16 datasets and seven LLMs. It beats the strongest baseline in 18 of 20 model-task combinations (Table 1). I recomputed the averages in Table 1 and Table 2 from the per-dataset tables in the appendix, and they match.
- The analysis goes beyond the main table. There is a per-component ablation (Table 2). There are results for retrieval with the semantic branch, the domain branch, and both together (Fig. 4). The paper also looks at how the gain changes with source-domain heterogeneity (Fig. 5). Results are reported for every dataset, including the cases where DRIFT loses.
Critical weaknesses
- The method is complex, but the paper does not show that this complexity is needed. DRIFT stacks five losses (classification, domain, adversarial, orthogonality, contrastive), plus prototype scoring. Removing any one component costs at most 1.16 points on Llama3.2-1B and at most 2.47 points on Qwen3-1.7B (Table 2). The full method is 10.83 and 3.54 points above plain kNN on these two models (Table 1). So most of the gain cannot be explained by any of the components the paper discusses. Fig. 4 points the same way. Retrieving with the domain branch z_d, which the method is meant to discard, still beats all four baselines on Qwen3-1.7B. On Llama it also beats kNN, DICL and UCS. The simplest explanation is that a supervised retriever helps by itself, not that removing domain information helps. The ablation is also run only on the two smallest models (1B and 1.7B). On the strong API models (Qwen3.5-Plus, DeepSeek-V4-Flash) the comparison is only against CCL, with 200 samples per domain, and the margin shrinks to 3.89 and 1.77 points. It is not clear whether these components still matter once the base model gets stronger. The control I need to see is a retriever trained with only the classification loss, run on both a small LLM and a strong LLM. If the authors can show that the full design is necessary, I would consider raising my score.
- Part of the evaluation does not measure retrieval quality. For NLI, the premise and hypothesis are concatenated first and then truncated to 100 tokens (dataloader.py). On ContractNLI the median premise length is about 1,400 words, so the hypothesis is cut off in every test sample. The largest effect in Fig. 1 (−17.1 on Qwen ContractNLI) and the main ablation effect on NLI (Table 10) both come from this dataset. The paper reports only accuracy. On CivilComments (92% benign), all methods, including fine-tuning, fall below majority-class accuracy.
- Related work. The OOD generalization part of Sec. 2 cites nothing newer than 2022, and the methods it cites are all from 2016-2021. Please add recent work on separating semantic and domain information, for example ProS (Fang et al., CVPR 2024), Disentangled Prompt Representation for Domain Generalization (Cheng et al., CVPR 2024) and DePro (Chen et al., SIGIR 2025).
Questions for authors
-
How much does a retriever trained with only the classification loss (same backbone, same prototype scoring) achieve? Please report it on at least one small LLM and one strong LLM. If it comes close to DRIFT, the paper should say so.
-
Please run the component ablation on Qwen3.5-Plus or DeepSeek-V4-Flash. Do the components still change the results?
-
After truncation on ContractNLI, can the model still see the hypothesis? Please report macro-F1 in all tables and add a majority-class row.
2 comments
I thank the authors for the additional experiments. The response resolves my main question in part. A retriever trained with only the classification loss falls 7.11 points below DRIFT on Llama3.2-1B and 3.32 points below it on Llama3.1-8B, so supervised retriever training alone does not explain the gain. This is the control I asked for, and it supports the method as a whole. Two concerns remain. First, the response does not show that each component is needed. On Llama3.2-1B, removing any single component costs at most 1.16 points in Table 2, while removing all the added training objectives costs 7.11, so the components appear largely interchangeable and a simpler variant may do as well. On the strong models the new ablation covers only sentiment analysis with 200 samples per dataset. On Qwen3.5-Plus the full method is ahead of each ablated variant by 0.25 to 0.75 points, which is 2 to 6 of 800 samples, and it is the best variant on one of the four datasets. Second, the corrected ContractNLI results weaken the NLI evidence. DRIFT is now ahead of kNN on one of the four models. If I replace only the ContractNLI column in the original per-dataset table, the NLI gain over kNN on Qwen3-1.7B falls from 3.02 to 0.02 points. The response does not rerun Fig. 1 or the NLI ablation, whose largest effects came from this dataset, and the macro-F1 table compares DRIFT only with CCL. I also could not find a revised PDF, so I cannot check how these results enter the paper. I keep my rating at 2.
Deeeeeeeeeear reviewer, has my response cleared up your concerns? Please drop by and check out my reply when you have time🙏