- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of selecting data mixtures for domain-adaptive instruction tuning under a fixed sample budget. The authors point out that two commonly used heuristics each rest on an untested assumption: the count heuristic (the more task types, the better) assumes that transfer is never negative, while the similarity heuristic (selecting the sources most similar to the target) assumes that transfer is symmetric. By generating all eight task types from a single corpus, withholding the target task during training, and probing with roughly 94 balanced mixtures, the authors fit an additive linear data model to obtain a signed and directed "transfer map."
Critical strengths
-
The problem is important and clearly motivated. Choosing the task mixture for synthetic supervision data is a real pain point in domain adaptation, where every trial-and-error attempt costs a fine-tuning run.
-
The core findings are crisp. Examples include the negative transfer of ERR -> MHOP, the differing directions of DEF -> MHOP versus MHOP -> DEF, and the fact that an interfering source can cancel the benefit of the other sources — results that offer direct takeaways for practitioners.
Critical weaknesses
-
All scores come from an automated rule, and the gold answers are generated by GPT-5-mini. I recommend adding at least a small-sample human evaluation or stricter format controls, and summarizing in the main text how this affects the main conclusions.
-
The cost to non-target tasks calls for a more practical discussion. The paper reports that the map-selected mixture scores 2.40 pp lower on average on the non-target macro. Real deployments often require balancing multiple objectives, whereas the current rule selects for a single target. The authors should provide a Pareto frontier, constrained selection, or trade-off results across multiple targets — or at least discuss when the selection rule should not be used.
-
A mechanistic explanation is absent. Once surface-level explanations such as token volume and answer length are ruled out, there is no diagnostic analysis at all of "what makes a task type an interferer" (e.g., gradient conflicts or representation-level interference). Leaving this as an open question is understandable, but it keeps the method at the level of "measurement," and practitioners cannot tell in advance whether running a probe is worthwhile on a new corpus.
-
The selector baselines could be stronger. The paper does not compare against gradient-influence methods such as LESS that "measure benefit rather than resemblance," and the comparison with the embedding-similarity selector is not cost-normalized — similarity selection costs zero training runs, whereas the transfer map requires dozens of fine-tuning runs. Showing at what probe budget the transfer-map route beats the free baseline would make the practical case more complete.
-
The controlled-corpus setup does not yet separate task type, passage content, and dose. The passage sets behind different task types overlap at a Jaccard of only 0.51–0.53; narrowing the mixture increases the number of examples per retained source; and some pools require repeated sampling. The check that answer length does not correlate with the coefficients is no substitute for these two controls. The cross-scale and cross-family correlations are valuable evidence of stability, but all models share the same generation and scoring protocol, and the second corpus changes document granularity, pool size, and evaluation size all at once. This is insufficient to attribute the effects to task format itself, or to support the dichotomous conclusion that transfer is a property of the data rather than of the model.
Questions for authors
-
Have you conducted any small-scale human evaluation to calibrate the scoring rule? Even a manual correctness judgment on 100 sampled generated answers from each of 2–3 predicted cells would substantially mitigate Weakness 1.
-
Under dose-matched controls (with the number of examples per source held fixed and the budget growing with d), does the directional asymmetry between DEF→MHOP and MHOP→DEF still hold?
-
Why did you choose unregularized OLS? Would shrinkage estimation of β (ridge or Bayesian) further reduce the variance of the 24-run probe and improve stability at small probe budgets?
-
How sensitive is the selected source set to τ? Would a transfer map fitted at τ=0.80 or 0.90 select the same source subsets?
-
To what extent is the cross-family agreement (r=0.88) driven by the shared generator and shared scoring rule? Does the cross-family consistency survive re-estimation with a different embedding model or with human scoring?
-
Under what conditions does the transfer-map route outperform the alternative of "spending the probe budget directly on generating training data of the target's own task type" (the Bonito-style route)? This seems to be the most natural alternative for practitioners, yet no such comparison appears in the paper.
2 comments
I thank the authors for the response. However, my main concerns aren't resolved. Therefore, I maintain my rating.
Changes according to oczQ's review
Thank you for the thorough and insightful review. Most of the new experiments came from your questions.
W1 / Q1. Human evaluation.
We did not add a human evaluation; Section 8 lists it as a limitation and future work. Our effects are differences of a few points, each measured on about 2,000 answers per neuroscience target at three seeds. A sample of 100 hand-graded answers would not reliably resolve differences that small.
We therefore added an independent second metric, an LLM judge (Gemma 4 31B), and refitted the map to it. The main trends hold: source identity predicts accuracy better than mixture size, and maps agree across model sizes. Mixtures selected from the judge-scored map beat the uniform mixture in all six large-model cases (three seeds).
→ Sections 7.1 and 8; Appendix H, Tables 14 and 16.
W2. Cost to non-target tasks.
We agree that selecting for one target has trade-offs in deployment. Averaged over the seven non-target tasks, the judge-selected mixtures differ from the uniform mixture by −3.1 to +3.9 pp. The clear loss falls on critiques when they are not the target, since the uniform mixture trains on them directly. Section 7 now notes that balancing several targets requires constrained selection; we did not build a Pareto frontier.
→ Section 7; Appendix H, "Cost and sensitivity".
W3. Mechanism.
We found one concrete mechanism: a model trained with the target withheld copies the answer format of its sources. Without multi-hop questions among the sources, about half of the critique answers restate the question.
In a format control with one example answer in the prompt, the gap to the uniform mixture under the judge closes on Qwen3-32B. It does not close on Mistral-24B, which keeps another source's format. We did not run gradient-conflict or representation-level diagnostics. Predicting which task type will interfere on a new corpus remains open.
→ Appendix H, "Why the selected sources differ".
W4. LESS and probe cost.
We did not run LESS; it scores single examples, while the transfer map scores task types. Similarity selection requires no training runs; the map needs a 24-run probe shared by all targets (11 to 22 GPU-hours).
Over three seeds under the embedding rule, the rule-scored map stays ahead of the embedding-similarity selector in five of six cases and ties with TaskShop, which uses the same probe. Under the judge, the judge-scored map beats both in all six.
With half the probe (12 runs), the judge-scored 8B map still drops its interferer in nearly every random subset, though the rest of the mixture often changes. We did not train those mixtures, so the break-even probe budget remains open.
→ Section 2; Appendix H, "Other selectors" and "Cost and sensitivity".
W5 / Q2. Passages, dose-matched control, and data vs. model.
Under the embedding rule, the map's mixture still beats the uniform mixture at equal examples per source in all four tested cases.
The passage-matched probe uses only passages shared by all four open task types and repeats no example. Under the embedding rule, at 600 examples per source (so the budget grows with the number of sources), DEF→MHOP is +6.70 pp [5.70, 7.73] and MHOP→DEF is −0.67 pp [−1.34, +0.04]. The directions keep their signs, though the negative effect cannot be distinguished from zero.
Section 7 now states that "a property of the data" means stability across models under one shared training and evaluation protocol. It also notes that PubMed changes several factors at once.
→ Sections 7 and 8; Appendix I, Tables 18 and 19.
Q3 / Q4. Unregularized OLS, ridge, and τ.
We used unregularized OLS as the simplest choice, since the rule reads only the signs. Ridge (penalty 0.1) on the judge-scored maps raises out-of-sample R² at all four sizes and changes the selected source subset in half of the cases, but every selection still drops the interferer.
The selection is sensitive to τ: at 0.80 or 0.90 it changes in most cases.
→ Appendix H, "Cost and sensitivity".
Q5. Shared generator and shared scoring rule.
We cannot rule them out as a driver of r = 0.88 (see W5). We had not kept the large models' probe answers, so we could not re-score that correlation. We tested an LLM judge rather than a second embedding model or human scoring. Under the judge, maps at four Qwen3 sizes agree, and mixtures selected at 8B beat the uniform mixture on both Qwen3-32B and Mistral-24B, a different family.
→ Section 7.1; Appendix H, Table 16.
Q6. Direct target supervision (the Bonito-style route).
Direct target supervision works better for critiques: with 2,000 or more examples it beats every mixture that withholds critiques. For WHY and MHOP it stays near or below the untuned model. These are single-seed runs. Section 8 now clarifies that we withhold the target to measure transfer. When target data are available, direct supervision is an option.
→ Section 8; Appendix H, "Cost and sensitivity".