- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies reliable batch construction for multi-domain RLVR. By the fact that individually useful prompts may interact poorly when trained in a batch, they propose TaylorMix. Experiments on multiple taskswith two Qwen backbones show consistent gains among baselines. Overall, the work offers a useful batch-level perspective on RLVR data selection.
Critical strengths
-
The paper identifies a practically relevant problem in multi-domain RLVR and proposes a clear two-level batch construction strategy that jointly considers domain allocation and prompt interactions.
-
The experiments cover multiple capability categories and two Qwen backbones, with consistent gains over the compared baselines and useful ablations for the main components.
Critical weaknesses
The two theorems mainly establish properties of the softmax objectives defined by the method, but do not show that the proposed cost reflects learning process or that the geometric penalty measures update status.
Questions for authors
-
Tab. 1 marks several results as statistically significant despite ties or very small gaps (e.g., AIME24: 25.83 vs. 25.83). Please clarify the statistical test, number of seeds, and how a tie can yield p < 0.05.
-
In Tab. 3, removing individual components causes very large drops on Arena-Hard, even below the untrained Base, while Full nearly matches Vanilla GRPO. Are these results from a single run?
-
Thm. 2 guarantees a lower expected composition penalty but does not necessarily imply higher expected utility. Please clarify what it establishes about the claimed value–composition balance beyond the entropy-regularized objective used to define the sampler.
6 comments
Q1: Tab. 1 marks several results as statistically significant despite ties or very small gaps (e.g., AIME24: 25.83 vs. 25.83). Please clarify the statistical test, number of seeds, and how a tie can yield p < 0.05.
We are sorry for the confusion. The original table reports a tie at 25.83. The annotation does not indicate a significant improvement over the method tied at that score. The test compares TaylorMix with AC-ODM, the strongest baseline with a non-tied score, using five independent training runs per method. We have corrected the table note and added the comparison and statistical details to Appendix D.2.
Training runs and standard deviations. Each method is trained independently with five different random seeds. Each resulting model is evaluated using Avg@8 on the 30 AIME24 problems, producing one seed-level score. We then compute the mean and sample standard deviation across the five seeds:
| Method | Independent training seeds | Avg@8: mean ± sample standard deviation |
|---|---|---|
| TaylorMix | 5 | 25.83 ± 0.29 |
| AC-ODM | 5 | 23.75 ± 0.66 |
Scores are percentages, and standard deviations are in percentage points. We compute the sample standard deviation with denominator . It measures variation across five independently trained models. The eight samples per problem contribute to each model's seed-level Avg@8 score. The statistical test uses training seeds as its units.
Significance test. We compare the methods' mean scores using a two-sided Welch's t-test, with training-seed scores as the statistical units. The result is , below 0.05.
Across these five seeds, TaylorMix improves on AC-ODM by an average of 2.08 percentage points. Appendix D.2 reports the means, standard deviations, and test results separately from the original table entries. This makes the comparison and supporting data explicit for each statistical conclusion.
Q2: In Tab. 3, removing individual components causes very large drops on Arena-Hard, even below the untrained Base, while Full nearly matches Vanilla GRPO. Are these results from a single run?
We trained Full and each ablation three times. The table reports each run's Arena-Hard and Arena-Creative scores, plus the mean and sample standard deviation for each dataset. This allows the performance drops to be compared with run-to-run variation:
| Configuration | Arena-Hard run 1 | Arena-Hard run 2 | Arena-Hard run 3 | Arena-Hard mean ± sample SD | Arena-Creative run 1 | Arena-Creative run 2 | Arena-Creative run 3 | Arena-Creative mean ± sample SD |
|---|---|---|---|---|---|---|---|---|
| Full TaylorMix | 22.05 | 22.70 | 23.65 | 22.80 ± 0.80 | 37.67 | 36.30 | 37.67 | 37.21 ± 0.79 |
| Without within-domain candidate selection | 9.25 | 10.28 | 10.38 | 9.97 ± 0.63 | 15.47 | 14.60 | 13.77 | 14.61 ± 0.85 |
| Without self-quadratic contribution (originally “without low-cost sketch abstraction”) | 10.70 | 10.65 | 9.87 | 10.41 ± 0.47 | 21.07 | 22.47 | 22.60 | 22.05 ± 0.85 |
| Without companion interactions (original No-curvature) | 16.77 | 16.83 | 16.50 | 16.70 ± 0.18 | 27.37 | 28.30 | 29.20 | 28.29 ± 0.92 |
Means and sample standard deviations are calculated separately for each dataset from its three scores. Scores are percentages, and standard deviations are in percentage points; the sample standard deviation uses the denominator .
Significance tests. We compare Full with each ablation using two-sided Welch's -tests. Each training run provides one score, and each group has . The -values for the two datasets are:
| Comparison with Full TaylorMix | Arena-Hard | Arena-Creative |
|---|---|---|
| Without within-domain candidate selection | ||
| Without self-quadratic contribution (originally “without low-cost sketch abstraction”) | ||
| Without companion interactions (original No-curvature) |
Our main experiments also show uneven gains across capabilities during multi-domain training. On Qwen2.5-1.5B, AC-ODM achieves a General mean of 8.17, above Vanilla GRPO's 5.88. Yet its Code Generation score is 0.75, below Vanilla GRPO's 2.75. Its HumanEval score of 45.73 is also below Base's 48.17. Gains in some capabilities can therefore accompany losses in others. Each capability needs to be evaluated separately.
Prior work reports similar learning imbalances and capability degradation. DISCO finds that GRPO favors dominant domains when domain distributions are imbalanced. Other domains can degrade even with within-group advantage normalization [1]. RECAP observes that reasoning-focused reinforcement learning can reduce existing general capabilities below Base [2]. OSOL reports cross-domain interference in multi-domain RL. It also reports an ablation that falls below Base after task weights are changed [3]. These studies help explain cross-domain capability changes. They also motivate examining training rewards alongside individual capability scores.
Our domain-allocation mechanism aims to mitigate this cross-domain learning imbalance. Target relevance and historical rewards guide the allocation of training slots. KL regularization limits departures from the reference mixture. This responds to current feedback while retaining learning opportunities for other domains. Domain allocation determines how many slots each domain receives. Within-domain selection determines which samples fill them. Together, they construct the training batch.
Full scores 22.80 on Arena-Hard, comparable to Vanilla GRPO's 22.79. Each ablation removes one component and retains the other selection mechanisms. This changes the selected training samples and their composition. The three-run results show both the resulting performance drops and the variation across runs.
References
[1] Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, and Furong Huang. 2025. DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data. In Findings of the Association for Computational Linguistics: EMNLP 2025.
[2] Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, and Deren Lei. 2025. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models. arXiv preprint arXiv:2510.21978.
[3] Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Guojun Yin, Wei Lin, and Ran He. 2026. One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control. arXiv preprint arXiv:2609.06469.
Thank you for recognizing the two-level batch construction strategy and multitask experiments. We address your comments on the theory and reliability of the results in three parts. First, we explain the selection properties and value–composition trade-off established by the theorems. We then clarify the AIME24 comparison and statistical test. Finally, we report three-run results and ablation tests on both Arena datasets.
We regret that the original presentation did not make the meaning and scope of our theoretical guarantees clear. The two theorems characterize domain allocation and composition-aware selection, respectively. Controlled experiments further assess their training value. We have added explanatory annotations to the relevant equations and distinguish these two forms of evidence below.
The theory gives two guarantees. Domain allocation can improve its relevance- and feedback-based cost while retaining opportunities for every domain. Composition-aware selection reduces the expected next-set penalty relative to individual-score sampling.
(1) Theorem 1: Improving allocation cost while retaining learning opportunities.
Domain costs combine target alignment and historical rewards so that training quotas respond to task requirements and observed model performance. We solve:
\rho_t^\star = \arg\min_{\rho\in\Delta^{K-1}} \left\\{ \underbrace{\sum_c\rho_c\mathrm{Cost}_c}_{\substack{\text{cost combining target alignment} \\\\ \text{and historical feedback}}} + \underbrace{\tau_\rho\mathrm{KL}(\rho\\|q)}_{\substack{\text{penalty for departing from} \\\\ \text{the reference mixture}}} \right\\}.
Here, ρ denotes domain training proportions and q is a positive reference mixture. At a positive temperature, Theorem 1 guarantees an average combined cost no greater than that of the reference mixture. Every domain also retains a positive continuous share. Thus, allocation can improve based on current feedback while preserving cross-domain learning opportunities. This share guarantee applies before quotas are rounded to integers.
(2) Theorem 2: Accounting for sample composition beyond individual scores improves composition-aware selection.
Individual scores compare candidates using relevance and historical rewards; the geometric terms additionally evaluate how sample deviations reinforce or offset one another. Theorem 2 gives the following next-selection guarantee:
expected geometric penalty withcomposition-aware selectionEi∼p,R(B∪i)≤expected geometric penalty withindividual-score selectionEi∼p0,R(B∪i).
The inequality is strict when the resulting penalties differ across candidates. The conditional score also corresponds exactly to the full set-utility increment, so both individual scores and composition effects guide selection.
The original controlled experiments assess the training value of these properties. The KL comparison shows better realized domain coverage. Full–No-curvature changes only the companion-interaction term. It retains the same domain-allocation rule, individual-score construction, and self-quadratic contribution. Retaining companion interactions improves overall capabilities on both backbones. The recorded trajectories also show fewer large gradient spikes and fewer all-correct or all-wrong response groups. The theory establishes selection properties; the experiments support their contribution to training.