acceptodds
Under review as a conference paper at ICLR 2027

Can domain synergy mislead? Measurement confounds in multi-domain RLVR interaction estimation

Abstract

Multi-domain RLVR post-training mixes mathematical reasoning, coding, agentic tool use, and general knowledge in one training run, and particular domain combinations are often credited with synergy. We ask whether such synergy can be measured reliably. We audit the full coalition lattice of four RLVR domains (GOPO, Qwen3-1.7B), evaluating every coalition on a fixed source suite used for selection and an independent transfer suite standing in for deployment, and decompose the gains into Mobius interaction dividends. Uncorrected, the lattice shows significant higher-order interaction that grows with order. We show that this pattern is largely a measurement artifact. When a fixed total budget is split across a coalition's domains, each domain's training depends on coalition size, which produces spurious higher-order dividends, and Mobius sums amplify training noise with order. After correcting for the budget split and accounting for seed variance, little higher-order interaction remains on the transfer suite, while the source suite is not fully resolved. The practical risk lies elsewhere: the coalition that looks best on the selection suite is often not the best on the deployment suite, and a single lattice cannot reliably identify the best mixture. We also examine a curvature-based account of cross-suite transfer and propose GIFT, an adaptive design for choosing which coalitions to train.

Primary area:
reinforcement learning
Keywords:
RLVR, Multi-domain Reinforcement Learning, Data mixtures, LLM post-training, Higher-order interaction, Mobius transform, Bayesian experimental design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8Q93OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether the domain synergy observed in multi-domain reinforcement learning with verifiable rewards (RLVR) reflects genuine interactions between domains and whether such synergy can further improve model training performance. The paper also examines the transferability of domain interactions across different evaluation distributions. It finds that interaction models can support domain combination selection reasonably well under the current experimental setting, while the optimal combination identified on the source evaluation suite may not reliably transfer to the test suite.

Critical strengths
  1. The paper examines a relevant measurement issue in multi-domain RLVR. It highlights how training-budget allocation may influence domain interactions and capability improvements, and provides a mathematical analysis of this effect.

  2. The paper includes several empirical analyses to support its observations. The authors consider different training-budget allocations, statistical correction methods, and evaluation settings. Although the experimental scale is relatively limited and the findings are not consistently reproduced across settings, these analyses provide some preliminary evidence for the identified measurement issue.

Critical weaknesses

1.The paper suffers from serious clarity and presentation issues. The manuscript contains numerous technical terms and expressions that are introduced without sufficient explanation and are not necessarily standard terminology in the field, such as "significant order-3 and order-4 interaction" and "D-optimal rule." Moreover, the research motivation and discussion of existing work are particularly difficult to follow. Overall, I find the manuscript insufficiently polished and closer to a preliminary draft than a publication ready paper.

  1. Several key claims lack sufficient empirical support. The experimental evidence does not adequately substantiate some of the paper's broader conclusions. First, the main findings are established primarily in a four domain setting using a small scale model, while the additional model and algorithm configuration fails to reproduce the same interaction patterns. Second, the curvature analysis does not directly evaluate the reward weighted Hessian quantities underlying the theoretical formulation. As a result, the experiments do not directly validate the theoretical analysis, leaving a substantial gap between the theoretical arguments and empirical evidence.
Questions for authors

See the weakness

Reviewer_dtqyOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper conducts a paired, fixed-regime audit over the full coalition lattice of four RLVR domains, holding a source suite and an independent transfer suite fixed throughout. Its central finding is a measurement confound: proportional per-coalition budget-splitting ties each domain’s update to coalition size, so even a non-interacting null can carry nonzero Möbius (Harsanyi) dividends. The authors examine this confound using single-domain budget-response curves, a three-domain lattice at fixed per-domain budget, and a seed-and-item bootstrap. In the GOPO/Qwen3-1.7B setting, at most one order-≥3 dividend survives multiplicity correction on transfer under these routes, while source remains unresolved. A lattice built by merging single-domain experts reproduces most uncorrected source survivors without joint training. A pairwise selection surrogate then loses essentially nothing, while the transfer-optimal coalition is not identifiable from a single-seed grid. The paper also measures the Fisher-proportional-curvature condition and finds it inconsistent with a single shared constant across the four domains. Finally, GIFT, a Bayesian experimental-design estimator, estimates the target dividends using fewer coalition queries; its A-optimal rule beats random selection at every tested budget, whereas the guarantee-carrying D-optimal rule does not.

Critical strengths

The paper identifies an important measurement confound in multi-domain RLVR interaction estimation. It shows that proportional budget-splitting across coalition members can produce nonzero higher-order Möbius dividends even under a non-interacting null, with an exact characterization through finite differences of single-domain budget-response curves. This provides a useful foundation for distinguishing budget-allocation effects from capability-level interaction when interpreting domain synergy.

The empirical audit uses informative, complementary controls to examine the interpretation of interaction signals. The full four-domain coalition lattice is evaluated on fixed source and transfer suites, and the analysis includes single-domain budget-response measurements, a fixed-per-domain-budget control, seed-and-item resampling, and expert merging without joint training. In particular, the expert-merge lattice reproduces six of the eight uncorrected source survivors, demonstrating that these signals need not require joint RLVR training.

Critical weaknesses

The selection analysis does not separate changes in the evaluation environment from ordinary ranking uncertainty. The paper reports nonzero source-to-transfer regret in 71.5% of resamples. However, the best coalition may also change when evaluated on different samples from the same suite, especially when coalition scores are close. As the authors acknowledge, there is no within-suite selection control. It therefore remains unclear how much additional selection error comes from changing evaluation suites, beyond the uncertainty already present within one suite.

The curvature experiment does not directly test the condition in the theoretical analysis. Proposition 1 concerns reward-weighted evaluation-risk curvature, whereas the experiment measures Fisher information. These are related but different quantities. Finding that the Fisher matrices are not proportional does not directly establish whether the curvature condition in Proposition 1 holds. Although the authors acknowledge this distinction, the connection between the curvature experiment and the theoretical explanation remains incomplete.

Questions for authors

Regarding coalition selection: Could the authors select a coalition on one subset of an evaluation suite and evaluate it on an independent subset of the same suite, using comparable sample sizes? How do the frequency and magnitude of regret in this within-suite control compare with those in the source-to-transfer setting?

Regarding curvature: Could the authors directly test the reward-weighted curvature condition in Proposition 1? If this is not feasible, what conclusions about that condition can be drawn from the existing Fisher-information measurements?

Reviewer_GaJwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how different training domains interact during multi-domain RLVR, and whether these interactions generalize to different evaluation benchmarks. The authors show that keeping the total training budget fixed across different domain combinations can introduce apparent higher-order interactions, even when the domains do not actually interact. They propose ways to account for this effect and find that most of the initially significant higher-order interactions are no longer significant after these corrections. They also investigate whether pairwise interactions are sufficient for choosing domain combinations and propose a method for estimating interactions with fewer training runs.

Critical strengths
  • Understanding how different training domains interact is an important problem, especially for post-training LLMs where we often want to combine several capabilities. I think the paper attempts to solve an interesting question about whether there are any domain interactions during pretraining.

  • The observation that fixing the total training budget can introduce interactions is interesting, and could be useful for interpreting results from other data mixture experiments.

  • I appreciate that the authors try to account for different sources of noise and validate their findings through multiple experimental settings.

Critical weaknesses
  • I am not fully convinced that the experiments show that higher-order interactions are unimportant after accounting for the training budget. The results seem sensitive to the correction method and how training variance is estimated, and some significant interactions remain. I think the paper provides evidence that the original measurements can be misleading, but the conclusions about the actual importance of higher-order interactions are less clear.

  • The conclusion that pairwise interactions are sufficient seems somewhat limited by the experimental setup. With only four domains, the pairwise model is fitted using 11 of the 15 possible domain combinations, and the best performing combinations are mostly pairs. I am not sure how much this tells us about settings with more domains, where higher-order interactions might be more important.

  • The paper is quite difficult to follow. There are several different research questions and experimental setups, and I found it difficult to identify the main contribution. I think the paper would benefit from focusing more on the central finding and simplifying the presentation.

Questions for authors
  1. The training budget is matched using the number of prompts rather than actual training FLOPs. Could differences in response lengths across domains affect the results?

  2. Would you expect the same findings to hold for larger models or different RL algorithms or more training domains?

  3. How would these findings change if we were optimizing continuous data mixture weights, rather than choosing which domains to include or exclude?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.