acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Batch Reliable? Balancing Multi-Domain RLVR with TaylorMix

Abstract

Multi-domain reinforcement learning with verifiable rewards (RLVR) can produce unstable policy updates even with self-generated responses. Our experiments identify batch composition as a key factor in update stability. Selecting high-scoring cases independently can overlook the risks of using them together, while concentrating training in favored domains can limit learning opportunities for other capabilities. These limitations motivate TAYLORMIX, a Taylor-motivated selector centered on balance. Before rollout, it balances domain priorities with continued opportunities across capabilities through KL-regularized allocation, and individual usefulness against composition risk through companion-dependent case selection. Our theoretical analysis characterizes this balance by quantifying the continuous-allocation trade-off and establishing conditional geometric-penalty reduction at a fixed selection prefix. Under matched training budgets, TAYLORMIX achieves the highest category means among the compared methods in mathematical reasoning, code, and general capability on two evaluated backbones. Ablations support both levels of balance, linking capability gains to preserved domain opportunities and geometric selection; the latter also yields fewer large gradient spikes and zero-advantage rollout groups in the observed trajectories. Our anonymous code is available at https://anonymous.4open.science/r/TaylorMix-D280.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards (RLVR), Data Mixing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zoNcOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reliable batch construction for multi-domain RLVR. By the fact that individually useful prompts may interact poorly when trained in a batch, they propose TaylorMix. Experiments on multiple taskswith two Qwen backbones show consistent gains among baselines. Overall, the work offers a useful batch-level perspective on RLVR data selection.

Critical strengths
  1. The paper identifies a practically relevant problem in multi-domain RLVR and proposes a clear two-level batch construction strategy that jointly considers domain allocation and prompt interactions.

  2. The experiments cover multiple capability categories and two Qwen backbones, with consistent gains over the compared baselines and useful ablations for the main components.

Critical weaknesses

The two theorems mainly establish properties of the softmax objectives defined by the method, but do not show that the proposed cost reflects learning process or that the geometric penalty measures update status.

Questions for authors
  1. Tab. 1 marks several results as statistically significant despite ties or very small gaps (e.g., AIME24: 25.83 vs. 25.83). Please clarify the statistical test, number of seeds, and how a tie can yield p < 0.05.

  2. In Tab. 3, removing individual components causes very large drops on Arena-Hard, even below the untrained Base, while Full nearly matches Vanilla GRPO. Are these results from a single run?

  3. Thm. 2 guarantees a lower expected composition penalty but does not necessarily imply higher expected utility. Please clarify what it establishes about the claimed value–composition balance beyond the entropy-regularized objective used to define the sampler.

6 comments

Thank you for recognizing the two-level batch construction strategy and multitask experiments. We address your comments on the theory and reliability of the results in three parts. First, we explain the selection properties and value–composition trade-off established by the theorems. We then clarify the AIME24 comparison and statistical test. Finally, we report three-run results and ablation tests on both Arena datasets.

Weakness: The two theorems mainly establish properties of the softmax objectives defined by the method, but do not show that the proposed cost reflects learning process or that the geometric penalty measures update status.

We regret that the original presentation did not make the meaning and scope of our theoretical guarantees clear. The two theorems characterize domain allocation and composition-aware selection, respectively. Controlled experiments further assess their training value. We have added explanatory annotations to the relevant equations and distinguish these two forms of evidence below.

The theory gives two guarantees. Domain allocation can improve its relevance- and feedback-based cost while retaining opportunities for every domain. Composition-aware selection reduces the expected next-set penalty relative to individual-score sampling.

(1) Theorem 1: Improving allocation cost while retaining learning opportunities.

Domain costs combine target alignment and historical rewards so that training quotas respond to task requirements and observed model performance. We solve:

\rho_t^\star = \arg\min_{\rho\in\Delta^{K-1}} \left\\{ \underbrace{\sum_c\rho_c\mathrm{Cost}_c}_{\substack{\text{cost combining target alignment} \\\\ \text{and historical feedback}}} + \underbrace{\tau_\rho\mathrm{KL}(\rho\\|q)}_{\substack{\text{penalty for departing from} \\\\ \text{the reference mixture}}} \right\\}.

Here, ρ\rho denotes domain training proportions and qq is a positive reference mixture. At a positive temperature, Theorem 1 guarantees an average combined cost no greater than that of the reference mixture. Every domain also retains a positive continuous share. Thus, allocation can improve based on current feedback while preserving cross-domain learning opportunities. This share guarantee applies before quotas are rounded to integers.

(2) Theorem 2: Accounting for sample composition beyond individual scores improves composition-aware selection.

Individual scores compare candidates using relevance and historical rewards; the geometric terms additionally evaluate how sample deviations reinforce or offset one another. Theorem 2 gives the following next-selection guarantee:

Ei∼p,R(B∪i)⏟expected geometric penalty withcomposition-aware selection≤Ei∼p0,R(B∪i)⏟expected geometric penalty withindividual-score selection.\underbrace{\mathbb E_{i\sim p}\\, \mathcal R(\mathcal B\cup\\{i\\})}_{\substack{\text{expected geometric penalty with} \\\\ \text{composition-aware selection}}} \le \underbrace{\mathbb E_{i\sim p^0}\\, \mathcal R(\mathcal B\cup\\{i\\})}_{\substack{\text{expected geometric penalty with} \\\\ \text{individual-score selection}}}.

The inequality is strict when the resulting penalties differ across candidates. The conditional score also corresponds exactly to the full set-utility increment, so both individual scores and composition effects guide selection.

The original controlled experiments assess the training value of these properties. The KL comparison shows better realized domain coverage. Full–No-curvature changes only the companion-interaction term. It retains the same domain-allocation rule, individual-score construction, and self-quadratic contribution. Retaining companion interactions improves overall capabilities on both backbones. The recorded trajectories also show fewer large gradient spikes and fewer all-correct or all-wrong response groups. The theory establishes selection properties; the experiments support their contribution to training.

Q1: Tab. 1 marks several results as statistically significant despite ties or very small gaps (e.g., AIME24: 25.83 vs. 25.83). Please clarify the statistical test, number of seeds, and how a tie can yield p < 0.05.

We are sorry for the confusion. The original table reports a tie at 25.83. The annotation does not indicate a significant improvement over the method tied at that score. The test compares TaylorMix with AC-ODM, the strongest baseline with a non-tied score, using five independent training runs per method. We have corrected the table note and added the comparison and statistical details to Appendix D.2.

Training runs and standard deviations. Each method is trained independently with five different random seeds. Each resulting model is evaluated using Avg@8 on the 30 AIME24 problems, producing one seed-level score. We then compute the mean and sample standard deviation across the five seeds:

MethodIndependent training seedsAvg@8: mean ± sample standard deviation
TaylorMix525.83 ± 0.29
AC-ODM523.75 ± 0.66

Scores are percentages, and standard deviations are in percentage points. We compute the sample standard deviation with denominator n−1=4n-1=4. It measures variation across five independently trained models. The eight samples per problem contribute to each model's seed-level Avg@8 score. The statistical test uses training seeds as its units.

Significance test. We compare the methods' mean scores using a two-sided Welch's t-test, with training-seed scores as the statistical units. The result is p=0.000898p=0.000898, below 0.05.

Across these five seeds, TaylorMix improves on AC-ODM by an average of 2.08 percentage points. Appendix D.2 reports the means, standard deviations, and test results separately from the original table entries. This makes the comparison and supporting data explicit for each statistical conclusion.

Q2: In Tab. 3, removing individual components causes very large drops on Arena-Hard, even below the untrained Base, while Full nearly matches Vanilla GRPO. Are these results from a single run?

We trained Full and each ablation three times. The table reports each run's Arena-Hard and Arena-Creative scores, plus the mean and sample standard deviation for each dataset. This allows the performance drops to be compared with run-to-run variation:

ConfigurationArena-Hard run 1Arena-Hard run 2Arena-Hard run 3Arena-Hard mean ± sample SDArena-Creative run 1Arena-Creative run 2Arena-Creative run 3Arena-Creative mean ± sample SD
Full TaylorMix22.0522.7023.6522.80 ± 0.8037.6736.3037.6737.21 ± 0.79
Without within-domain candidate selection9.2510.2810.389.97 ± 0.6315.4714.6013.7714.61 ± 0.85
Without self-quadratic contribution (originally “without low-cost sketch abstraction”)10.7010.659.8710.41 ± 0.4721.0722.4722.6022.05 ± 0.85
Without companion interactions (original No-curvature)16.7716.8316.5016.70 ± 0.1827.3728.3029.2028.29 ± 0.92

Means and sample standard deviations are calculated separately for each dataset from its three scores. Scores are percentages, and standard deviations are in percentage points; the sample standard deviation uses the denominator n−1=2n-1=2.

Significance tests. We compare Full with each ablation using two-sided Welch's tt-tests. Each training run provides one score, and each group has n=3n=3. The pp-values for the two datasets are:

Comparison with Full TaylorMixArena-Hard ppArena-Creative pp
Without within-domain candidate selection4.164×10−54.164\times10^{-5}4.856×10−64.856\times10^{-6}
Without self-quadratic contribution (originally “without low-cost sketch abstraction”)1.133×10−41.133\times10^{-4}2.342×10−52.342\times10^{-5}
Without companion interactions (original No-curvature)4.211×10−34.211\times10^{-3}2.443×10−42.443\times10^{-4}

Our main experiments also show uneven gains across capabilities during multi-domain training. On Qwen2.5-1.5B, AC-ODM achieves a General mean of 8.17, above Vanilla GRPO's 5.88. Yet its Code Generation score is 0.75, below Vanilla GRPO's 2.75. Its HumanEval score of 45.73 is also below Base's 48.17. Gains in some capabilities can therefore accompany losses in others. Each capability needs to be evaluated separately.

Prior work reports similar learning imbalances and capability degradation. DISCO finds that GRPO favors dominant domains when domain distributions are imbalanced. Other domains can degrade even with within-group advantage normalization [1]. RECAP observes that reasoning-focused reinforcement learning can reduce existing general capabilities below Base [2]. OSOL reports cross-domain interference in multi-domain RL. It also reports an ablation that falls below Base after task weights are changed [3]. These studies help explain cross-domain capability changes. They also motivate examining training rewards alongside individual capability scores.

Our domain-allocation mechanism aims to mitigate this cross-domain learning imbalance. Target relevance and historical rewards guide the allocation of training slots. KL regularization limits departures from the reference mixture. This responds to current feedback while retaining learning opportunities for other domains. Domain allocation determines how many slots each domain receives. Within-domain selection determines which samples fill them. Together, they construct the training batch.

Full scores 22.80 on Arena-Hard, comparable to Vanilla GRPO's 22.79. Each ablation removes one component and retains the other selection mechanisms. This changes the selected training samples and their composition. The three-run results show both the resulting performance drops and the variation across runs.

References

[1] Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, and Furong Huang. 2025. DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data. In Findings of the Association for Computational Linguistics: EMNLP 2025.

[2] Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, and Deren Lei. 2025. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models. arXiv preprint arXiv:2510.21978.

[3] Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Guojun Yin, Wei Lin, and Ran He. 2026. One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control. arXiv preprint arXiv:2609.06469.

Reviewer_wpevOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies pre-rollout prompt selection for multi-domain RLVR with GRPO. The authors argue that scoring prompts independently and mixing domains without regularization cause unstable updates and uneven progress across skills. Motivated by a second-order Taylor expansion of one-step loss improvement, they propose TaylorMix, a two-stage batch selector. At the domain level, TaylorMix minimizes an expected domain cost (combining prototype distance and reward history) with a KL penalty toward a reference mixture qq, then rounds the tilted softmax weights into integer quotas. Within each quota, it selects prompts sequentially without replacement using a conditional score Γi∣k\Gamma_{i\mid k} that subtracts a sketch-space quadratic penalty R(B)=12zB⊤GtzB\mathcal{R}(B) = \frac{1}{2} z_B^\top G_t z_B built from Gaussian projections of prompt residuals and an EMA Gram matrix. Experiments on Qwen3.5-2B and Qwen2.5-1.5B across Math, Code, and General tasks show steady gains.

Critical strengths
  • Selecting prompts before rollout is a practical way to save compute in RLVR. Splitting the problem into KL-regularized domain quotas followed by within-domain subset selection makes intuitive sense and adds very little wall-clock overhead (39.49s39.49\text{s} vs. 37.94s37.94\text{s} per case in Appendix E.2).
  • The prefix rescoring trick using cached sketch residuals ziz_i and an EMA Gram matrix GtG_t (Eq. 4) is lightweight and visibly reduces gradient spikes in Figure 3a.
Critical weaknesses
  1. While Eq. (D.7) motivates R(B)\mathcal{R}(B) via the parameter Hessian ut⊤∇2M(θt)utu_t^\top \nabla^2 \mathcal{M}(\theta_t) u_t, Eq. (3) uses the covariance GtG_t of static prompt sketches (Eqs. A.11-A.12). Minimizing zB⊤GtzBz_B^\top G_t z_B just pulls the domain mean residual toward pcp_c, ignores cross-domain terms since zc,t(0)=0z_{c,t}^{(0)}=0 resets per domain (Alg. 1, l.13), and ignores GRPO advantage signs.
  2. Curvature is subtracted twice without scale matching: SiS_i in Eq. (A.5) already subtracts normalized zi⊤Gtziz_i^\top G_t z_i clipped to [−a,a][-a,a] (Eq. A.8), and Eq. (4) subtracts unnormalized zi⊤Gtzi/(2(k+1))z_i^\top G_t z_i / (2(k+1)) again (l.743). Also, Sec. 6.1 and App. A.1 cite App. E.4 for selector settings (τρ,Ts,λ∗,a,∣At∣\tau_\rho, T_s, \lambda_*, a, |\mathcal{A}_t|) and ψcost,ψlearn\psi_{\text{cost}}, \psi_{\text{learn}} (Eqs. A.7, A.13), which Table 7 omits.
  3. Table 7 uses only B=6B=6 prompts across K=3K=3 domains (~2/domain) and 2 rollouts/prompt, limiting set interactions and contradicting l.1085 ("1 groups with 30 cases").
Questions for authors
  1. What are ψcost,ψlearn\psi_{\text{cost}}, \psi_{\text{learn}} (Eqs. A.7, A.13) and the missing settings in App. E.4, and how is R(B)\mathcal{R}(B) scaled to SiS_i in Eq. (4)?
  2. How can static prompt covariance GtG_t proxy ut⊤∇2M(θt)utu_t^\top \nabla^2 \mathcal{M}(\theta_t) u_t without advantage signs?

Above all, my major concern is the curvature proxy mismatch, I'm open to discuss.

6 comments

Thank you for recognizing the practicality, two-level structure, and low overhead of pre-rollout selection. We address your questions about the curvature proxy, score scales, and small-batch statistics below. We explain the proxy, provide selector settings, and clarify how the statistics are calculated. We also distinguish the roles of historical rewards, composition geometry, and GRPO updates.

The Hessian and the Pre-Rollout Composition Proxy

(1) Repeatedly evaluating candidate combinations using a large model's parameter Hessian is costly, so we use a low-cost geometric proxy. The Taylor expansion in Eq. (D.7) reveals the first-order gain and second-order composition effects of an update:

M(θ)−M(θ−ηu)=ηg⊤u⏟first-order gain−η22u⊤Hu⏟second-ordercomposition term−R3(u),H=∇2M(θ).M(\theta)-M(\theta-\eta u) = \underbrace{\eta g^\top u}_{\text{first-order gain}} - \underbrace{\frac{\eta^2}{2}u^\top H u}_{\substack{\text{second-order} \\\\ \text{composition term}}} -R_3(u), \qquad H=\nabla^2M(\theta).

Here, g=∇M(θ)g=\nabla M(\theta), uu is the batch-induced update direction, and R3R_3 is the higher-order remainder. The quadratic term contains both self-contributions and companion interactions. Thus, a sample's training value also depends on how it is combined with others. This motivates a composition-risk proxy in a low-dimensional representation space, which we use for pre-rollout selection.

(2) The proxy retains the structure of self-contributions and companion interactions, while enabling low-cost rescoring with cached representations. For a single anchor Gram matrix:

GA=1∣A∣∑a∈Azaza⊤,RA(B)=12zˉB⊤GAzˉB=12∣A∣∑a∈A(zˉB⊤za)2.G_A=\frac1{|A|}\sum_{a\in A}z_a z_a^\top, \qquad \mathcal R_A(\mathcal B) =\frac12\bar z_{\mathcal B}^{\top}G_A\bar z_{\mathcal B} =\frac1{2|A|}\sum_{a\in A}\left(\bar z_{\mathcal B}^{\top}z_a\right)^2.

This proxy measures the average squared alignment of the set's mean residual with the anchors. An EMA smooths these statistics to form GtG_t. The quadratic terms evaluate both a candidate's own contribution and how it complements the selected samples. We reuse cached residuals and Gram statistics during selection. Updating the set mean then allows us to rescore the remaining candidates.

Theorem 2 and its derivation link conditional scores exactly to proxy-utility increments. At the same selection state, the expected geometric penalty is no greater than under individual-score sampling. The derivation also bounds the proxy-utility gap to the best next selection. These properties support the proxy as a composition-aware selection criterion.

(3) We select prompts before rollout, so current rewards are not yet available. The subsequent GRPO update uses the corresponding signed learning signals. Ranking all candidates by their current response rewards would require rolling out and scoring the entire candidate pool. This would incur the generation cost that preselection is intended to save. We therefore use historical rewards and target relevance to assess individual value before rollout. The geometric proxy assesses composition. After rollout, current group-relative rewards determine the advantages used for the policy update.

With this design in place, we address the three specific questions about the set-mean constraint, cross-domain interactions, and advantage signs below.

W1: While Eq. (D.7) motivates R(B)R(B) via the parameter Hessian ut⊤∇2M(θt)utu_t^\top\nabla^2 M(\theta_t)u_t, Eq. (3) uses the covariance GtG_t of static prompt sketches (Eqs. A.11–A.12). Minimizing zˉB⊤GtzˉB\bar z_B^\top G_t\bar z_B just pulls the domain mean residual toward pcp_c, ignores cross-domain terms since zc,t(0)=0z_{c,t}^{(0)}=0 resets per domain (Alg. 1, l.13), and ignores GRPO advantage signs.

We address each of these three points in turn.

(1) “just pulls the domain mean residual toward pcp_c”

This term is intended to constrain the selected set's mean deviation from the domain prototype. It encourages the set mean to stay closer to the domain's representation center as measured by GtG_t. It provides geometric balance within a domain, while SiS_i assesses individual candidate value using relevance and historical rewards. The full objective combines both roles: it considers sample value and controls the samples' collective deviation. Proximity to the center is therefore not the sole selection criterion, nor must every sample be close to the center.

(2) “ignores cross-domain terms since zc,t(0)=0z_{c,t}^{(0)}=0 resets per domain (Alg. 1, l.13)”

Domains share geometric statistics. The weighted sum of within-domain penalties upper-bounds the full-batch geometric penalty. All domains use the same GtG_t, built from anchors across domains. Consider each domain represented in the selected batch. Let αc=∣Bc∣/∣B∣\alpha_c=|\mathcal B_c|/|\mathcal B| denote its batch proportion and zˉc\bar z_c the mean residual of its selected samples. The combined mean residual is ∑cαczˉc\sum_c\alpha_c\bar z_c. Since Gt⪰0G_t\succeq0, the function 12z⊤Gtz\tfrac12 z^\top G_tz is convex. We apply the same geometric proxy to the combined batch. Jensen's inequality then gives:

12(∑cαczˉc)⊤Gt(∑cαczˉc)⏟full-batchgeometric penalty≤∑cαc12zˉc⊤Gtzˉc⏟weighted sum ofwithin-domain penalties.\underbrace{\frac12 \left(\sum_c\alpha_c\bar z_c\right)^\top G_t \left(\sum_c\alpha_c\bar z_c\right)}_{\substack{\text{full-batch} \\\\ \text{geometric penalty}}} \le \underbrace{\sum_c\alpha_c\frac12\bar z_c^\top G_t\bar z_c}_{\substack{\text{weighted sum of} \\\\ \text{within-domain penalties}}}.

For fixed domain quotas, reducing the within-domain geometric penalties lowers this upper bound on the full-batch geometric penalty. We reset the prefix for each domain. Each candidate's composition contribution is then evaluated against the samples already selected in that domain. Adaptive quotas use target alignment and historical rewards to coordinate training weights across domains. Together, these two levels allocate cross-domain learning opportunities and control within-domain composition quality.

(3) “ignores GRPO advantage signs”

Historical rewards inform value assessment before rollout. Current advantage signs guide the policy update after rollout. At selection time, current responses and their group-relative rewards are not yet available. We compute SiS_i from historical feedback. The geometric terms evaluate the candidate's fit with selected samples. Although GtG_t is positive semidefinite, the interaction zi⊤Gtzˉkz_i^\top G_t\bar z_k can have either sign. It can therefore distinguish reinforcing from complementary residual directions.

After rollout, the unchanged GRPO procedure computes signed advantages from current within-group rewards and uses them to update the policy. Historical feedback guides value assessment, geometry guides composition before rollout, and current feedback guides the policy update. The original Full–No-curvature comparison and training-dynamics analysis further support the training value of companion interactions in pre-rollout selection.

W2: Curvature is subtracted twice without scale matching: SiS_i in Eq. (A.5) already subtracts normalized zi⊤Gtziz_i^\top G_t z_i clipped to [−a,a][-a,a] (Eq. A.8), and Eq. (4) subtracts unnormalized zi⊤Gtzi/(2(k+1))z_i^\top G_t z_i/(2(k+1)) again (l.743). Also, Sec. 6.1 and App. A.1 cite App. E.4 for selector settings (τρ,Ts,λ\*,a,∣At∣\tau_\rho, T_s, \lambda_\*, a, |\mathcal{A}_t|) and ψcost,ψlearn\psi_{\mathrm{cost}}, \psi_{\mathrm{learn}} (Eqs. A.7, A.13), which Table 7 omits.

Thank you for pointing out the error in Eq. (A.5), and we are sorry for the confusion it caused. The error was in the manuscript's description of the individual score. All experiments used only target relevance and historical reward feedback in SiS_i, each with weight 1/21/2. Geometric contributions enter through the set penalty. This correction changes only the manuscript, not the experimental procedure or results.

(1) Individual scores express candidate value.

The corrected Eq. (A.5) is:

Si=12Rel~i+12Learn~i.S_i=\frac12\widetilde{\mathrm{Rel}}_i +\frac12\widetilde{\mathrm{Learn}}_i.

The two signals are separately standardized over the original candidate window and clipped to [−a,a][-a,a]. We have marked the erroneously included normalized geometric term with a blue strikethrough. We have also corrected the main-text description and the weights reported in the manuscript.

(2) Self-quadratic contributions and companion interactions come from the same set penalty.

Let Qi=zi⊤GziQ_i=z_i^\top Gz_i and let Δi∣k\Delta_{i\mid k} denote the set-utility increment from adding candidate ii. Expanding the full set objective gives Eq. (D.4):

Γi∣k=Si⏟individual score−kk+1zi⊤Gzˉk⏟companioninteraction−Qi2(k+1)⏟self-quadraticcontribution=(k+1)Δi∣k−bk.\Gamma_{i\mid k} = \underbrace{S_i}_{\text{individual score}} - \underbrace{\frac{k}{k+1}z_i^\top G\bar z_k}_{\substack{\text{companion} \\\\ \text{interaction}}} - \underbrace{\frac{Q_i}{2(k+1)}}_{\substack{\text{self-quadratic} \\\\ \text{contribution}}} = (k+1)\Delta_{i\mid k}-b_k.

Here, bkb_k is the same for all candidates, so conditional scores and set-utility increments give the same ranking. The two geometric coefficients, k/(k+1)k/(k+1) and 1/[2(k+1)]1/[2(k+1)], follow from the set-mean update. The self-quadratic contribution remains at the first draw as Qi/2Q_i/2.

This correction preserves the increment derivation. Theorem 2 treats SiS_i as a fixed, finite score at the current selection state. It does not require a geometric term inside SiS_i. With the same SiS_i used in the set utility and conditional score, the expected-penalty comparison and one-step utility bound still hold.

(3) Scales and experimental settings.

The individual-score weights are (λrel,λlearn)=(1/2,1/2)(\lambda_{\mathrm{rel}},\lambda_{\mathrm{learn}})=(1/2,1/2), with clipping threshold a=3a=3. The set penalty has coefficient one in the utility. Its interaction and self-quadratic terms use the same raw Gram scale. We empirically measure QiQ_i at approximately 6.46. Within the individual score, normalization applies to the two value signals. The geometric terms enter the conditional score through the full objective above.

The original experiments use a uniform reference domain mixture. The domain-allocation and selection temperatures are τρ=1.0\tau_\rho=1.0 and Ts=0.7T_s=0.7. The geometric proxy uses a 64-dimensional residual projection and 128 fixed anchors per domain. An EMA updates the Gram statistics. Both reward mappings are identity functions. Appendix D.4 provides the full settings for normalization, representation refreshes, and historical-feedback updates.

Reviewer_8GsROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes TaylorMix, a pre-rollout data-selection method for multi-domain RLVR. It combines KL-regularized domain allocation with sequential, composition-aware selection within each domain, using cached representations and reward feedback. The analysis establishes properties of the continuous allocation and conditional reductions in a geometric surrogate penalty. Experiments on two Qwen backbones report improvements across mathematics, coding, and general capabilities, supported by component ablations and training-dynamics comparisons.

Critical strengths
  1. Practical integration of domain allocation and batch composition. TaylorMix explicitly accounts for how a candidate interacts with previously selected examples while preserving domain coverage. Performing this selection before rollout, without additional speculative generations or changes to GRPO, makes the approach practically relevant.

  2. Consistent empirical improvements across the evaluated settings. The method achieves the highest reported category means across all three capability families on both backbones. The allocation and geometry ablations provide additional evidence that both components contribute, supporting the usefulness of the combined design.

Critical weaknesses
  1. The benefit of companion-dependent selection is not clearly isolated. The selector includes both individual geometric penalties and a prefix-dependent interaction term. Tables 3–4 and Figure 3 support geometric selection overall, but the “without curvature” ablation does not clearly distinguish these contributions. Consequently, the improvements could partly reflect individual geometric filtering rather than the proposed interaction mechanism. This limits the evidence for the paper’s central claim that accounting for which examples are selected together provides additional value.

  2. The benefit of adaptive domain allocation beyond preserving coverage remains unclear. Table 2 primarily compares KL-regularized allocation against an unregularized alternative that loses domain coverage. This supports retaining learning opportunities across domains, but does not establish an advantage over a fixed, coverage-preserving mixture using the same within-domain selector. Although Domain2Vec and AC-ODM are included, those comparisons also change case selection. A controlled comparison is therefore missing for assessing the distinct contribution of the proposed allocation mechanism.

Questions for authors
  1. Which terms are removed in the “without curvature” variant? Does any reported comparison disable only the prefix-dependent interaction in Equation (4), while retaining all individual geometric penalties and the same domain allocation?

  2. With the within-domain selector held fixed, how does the proposed allocation compare against using the fixed reference mixture q throughout training? Such a comparison would help determine whether adaptive allocation adds value beyond maintaining domain coverage.

6 comments

Thank you for recognizing the practicality of pre-rollout selection. Your comments identify two questions for the ablations:

  • Do companion interactions add value when individual scores and self-quadratic contributions are retained?
  • Does adaptive domain allocation improve on a fixed reference mixture with the same within-domain selector?

These questions concern sample composition within domains and quota allocation across domains. We first clarify that the original No-curvature ablation removes only companion interactions. We then introduce No-self-score, which removes the entire individual score, and Fixed-q, which fixes the domain mixture.

W1 / Q1: Which terms are removed in the “without curvature” variant? Does any reported comparison disable only the prefix-dependent interaction in Equation (4), while retaining all individual geometric penalties and the same domain allocation?

The original “without curvature” variant, called No-curvature below, is the No-interaction control. It removes only the prefix-dependent cross term in Eq. (4). It retains the individual score SiS_i, the self-quadratic contribution, and the same adaptive domain-allocation rule. Full–No-curvature thus tests the added value of composition while retaining candidate value and self-quadratic contributions. The new No-self-score control tests the role of the entire individual score.

The full conditional score in Eq. (4) is:

Γi∣kFull=Si⏟individual score−kk+1zi⊤Gtzˉk⏟companioninteraction−zi⊤Gtzi2(k+1)⏟self-quadraticcontribution.\Gamma^{\mathrm{Full}}_{i\mid k} = \underbrace{S_i}_{\text{individual score}} - \underbrace{\frac{k}{k+1}z_i^\top G_t\bar z_k}_{\substack{\text{companion} \\\\ \text{interaction}}} - \underbrace{\frac{z_i^\top G_tz_i}{2(k+1)}}_{\substack{\text{self-quadratic} \\\\ \text{contribution}}}.

Here, SiS_i combines normalized relevance and historical reward feedback, each with weight 1/21/2. The geometric terms account for residual complementarity with selected samples and the candidate's own contribution to the set penalty.

The original No-curvature variant removes only the companion-interaction term −kk+1zi⊤Gtzˉk-\frac{k}{k+1}z_i^\top G_t\bar z_k, giving:

Γi∣kNo-curvature=Si−zi⊤Gtzi2(k+1).\Gamma^{\mathrm{No\text{-}curvature}}_{i\mid k} = S_i-\frac{z_i^\top G_tz_i}{2(k+1)}.

With the same candidate window and selection state, Full and No-curvature assign identical scores at the first draw. From the second draw onward, Full adjusts candidate priorities based on the selected samples. This original control therefore distinguishes a candidate's individual suitability from its suitability for the current composition.

We introduce No-self-score, which removes the entire SiS_i term from Eq. (4). It retains both companion interactions and self-quadratic contributions. Its conditional score is:

Γi∣kNo-self-score=−kk+1zi⊤Gtzˉk−zi⊤Gtzi2(k+1).\Gamma^{\mathrm{No\text{-}self\text{-}score}}_{i\mid k} = -\frac{k}{k+1}z_i^\top G_t\bar z_k -\frac{z_i^\top G_tz_i}{2(k+1)}.

This control tests whether relevance- and reward-based individual scores add value when set-level geometry is retained. It uses the same adaptive domain-allocation rule. Its scores and sampling probabilities can differ from Full from the first draw.

The original without low-cost sketch abstraction is a separate ablation. It removes only the self-quadratic contribution zi⊤Gtzi/[2(k+1)]z_i^\top G_tz_i/[2(k+1)], retaining SiS_i and companion interactions. Below, we label it “Without self-quadratic contribution” and retain its original name for reference.

We compare checkpoints at matched training steps. All arms start from the same initial model and use the same data, optimization, and reward settings. Each arm maintains its own online feedback and selector state. Full–No-curvature tests companion interactions; Full–No-self-score tests the entire individual score. Each comparison measures the score term's overall effect on subsequent training and capabilities. The results are reported below.

W2 / Q2: With the within-domain selector held fixed, how does the proposed allocation compare against using the fixed reference mixture q throughout training? Such a comparison would help determine whether adaptive allocation adds value beyond maintaining domain coverage.

The key to this comparison is to hold the within-domain selector fixed. We therefore add a Fixed-q control that changes only the allocation of training slots across domains while retaining the same within-domain selection rule.

Full updates domain costs using target alignment and historical feedback, with KL regularization yielding:

ρc,tFull=qcexp⁡(−Costc,t/τρ)∑jqjexp⁡(−Costj,t/τρ).\rho^{\mathrm{Full}}_{c,t} = \frac{q_c\exp(-\mathrm{Cost}_{c,t}/\tau_\rho)} {\sum_jq_j\exp(-\mathrm{Cost}_{j,t}/\tau_\rho)}.

Fixed-q instead directly uses the reference mixture throughout training:

ρc,tFixed-q=qc.\rho^{\mathrm{Fixed\text{-}q}}_{c,t}=q_c.

Both use the same quota-rounding and candidate-supply rules. Their within-domain selection rules are also identical: both retain individual scores, companion interactions, and self-quadratic contributions. The additional experiments use a uniform reference mixture. With six prompts per step, three domains, and sufficient candidate supply, Fixed-q assigns two slots to each domain. The configurations differ as follows:

ConfigurationDomain allocationIndividual score SiS_iSelf-quadratic contributionCompanion interaction
FullAdaptiveRetainedRetainedRetained
Original No-curvatureAdaptiveRetainedRetainedRemoved
No-self-scoreAdaptiveRemovedRetainedRetained
Without self-quadratic contribution (originally “without low-cost sketch abstraction”)AdaptiveRetainedRemovedRetained
Fixed-qFixed reference mixtureRetainedRetainedRetained

Each control changes the corresponding component, separating within-domain scoring from domain-level allocation.

Continuous shares are rounded into integer training quotas. A small positive share can therefore be rounded to zero. Full and Fixed-q have different realized domain coverage in these experiments. The results below reflect the overall effect of the allocation policy, including the resulting changes in domain exposure.

Original and Additional Experimental Results

The table combines the original results and additional controls. No-curvature removes companion interactions. The new No-self-score control removes the entire individual score. The original “without low-cost sketch abstraction” ablation removes the self-quadratic contribution. For each metric, methods are compared at matched training steps. Math and code results use their respective evaluation checkpoints. The code results for Full, No-self-score, and Fixed-q use the same generation and scoring protocol.

ConfigurationAIME24AIME25LCB Code ExecutionArena-Hard
Full25.8322.9227.7722.80
Original No-curvature15.4220.0020.4316.70
No-self-score20.5017.5026.2821.87
Without self-quadratic contribution (originally “without low-cost sketch abstraction”)24.5822.0817.1210.41
Fixed-q15.8315.4223.5722.23

Fixed-q improves Code and General performance but shows substantial forgetting in mathematics. Relative to the initial Base model, LCB Code Execution rises from 17.42 to 23.57 and Arena-Hard from 19.75 to 22.23. In contrast, AIME24 falls from 20.42 to 15.83 and AIME25 from 18.75 to 15.42.

No-self-score underperforms Full on all four metrics. This supports the value of relevance- and reward-based individual scores when set-level geometry is retained. Removing the entire SiS_i reduces AIME24 from 25.83 to 20.50 and AIME25 from 22.92 to 17.50. Code Execution falls from 27.77 to 26.28, and Arena-Hard from 22.80 to 21.87. The drops are larger in mathematics. These results support retaining individual value signals alongside the geometric evaluation of sample composition.

In summary, Full–No-curvature tests companion interactions. The new Full–No-self-score comparison tests the entire individual score. The original “without low-cost sketch abstraction” ablation tests the self-quadratic contribution. Full–Fixed-q compares allocation policies with the same within-domain selector. Together, these controls separate the components of the full selection mechanism.

Thank you for the clarifications and experiments.

For each metric, methods are compared at matched training steps. Math and code results use their respective evaluation checkpoints.

I may be misunderstanding this. Could you clarify whether the math, code, and general-capability results all come from the same checkpoint? If not, how do these results demonstrate improvements across domains in a single model?

Comments on the paper

2 comments

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

Dear Area Chair,

Thank you for guiding the discussion. We are grateful to the reviewers for their careful reading. All three recognized our contribution to batch construction for multi-domain RLVR: combining domain allocation with composition-aware prompt selection before rollout and demonstrating the practical value of this design empirically. Their questions prompted us to add more targeted comparisons and clarify the theoretical and empirical results. We summarize the main additions below.

  1. Component ablations (8GsR W1–W2 / Q1–Q2). We clarified that No-curvature removes only the companion interaction and added No-self-score (removing only the individual score) and Fixed-q (fixing the domain mixture while retaining the full within-domain selector). Full outperforms all three controls on all four metrics in Appendix D.5.
  2. Geometric proxy and learning dynamics (wpev W1 / Q2; zoNc W / Q3). The weighted sum of within-domain penalties upper-bounds the full-batch geometric penalty. GRPO still uses signed advantages from the current rollouts. At a fixed selection prefix, Theorem 2 guarantees no increase in the expected geometric penalty relative to individual-score sampling, and the appendix bounds the possible loss in proxy utility. Retaining companion interactions improves performance on both backbones, with fewer large gradient spikes in the observed trajectories.
  3. Implementation and measurement (wpev W2–W3 / Q1). We corrected the description of Eq. (A.5): individual scores use equally weighted target relevance and historical rewards, and geometry enters through the set penalty. Appendix D.4 documents the selector settings and identity reward mappings. Figure 3(b) aggregates 30 response groups across five consecutive steps.
  4. Evaluation checks (zoNc Q1–Q2; 8GsR follow-up). The AIME24 significance test compares TaylorMix with AC-ODM using five training seeds per method (two-sided Welch test, p = 0.000898). We reported three independent runs of the Arena ablations.

We appreciate the reviewers’ follow-up. 8GsR said the clarifications addressed their concerns and that they would raise their score to 3. wpev noted that their concerns were partly addressed and reported raising their score to 3. zoNc acknowledged the empirical contributions and kept their original rating. We hope this summary makes the additional results easy to locate. Thank you again for your time and consideration.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.