acceptodds
Withdrawn from ICLR 2027

Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training via Reinforcement Learning (RL) has enabled Large Reasoning Models (LRMs) to achieve strong performance in individual domain. However, real-world applications increasingly require general-purpose reasoners rendering strong performance across diverse domains. Mixed-domain post-training aims to achieve this goal by jointly training with mixed domain data, but this often induces capability compromise and degradation among different domains. Existing methods attribute this performance degradation to harmful cross-domain interactions and propose various strategies to mitigate them, but these strategies may also impede beneficial knowledge sharing across domains and in turn fail to match or surpass single-domain performance. To address this problem, we propose Multi-domain Contrastive Policy Optimization (MCPO), which uses contrastive learning to utilize both positive and negative cross-domain interactions for knowledge sharing and competition. Specifically, we partition each rollout generated by LRMs according to its underlying reasoning structures and use reasoning segments to capture these structures. We thus formulate positive and negative pairs of reasoning segments as mutually augmented examples, which provide supportive and competing signals for knowledge sharing. Subsequently, we design complementary contrastive objectives for cross-domain knowledge sharing and intra-domain knowledge consolidation, targeting compatibility across domains and discriminability within each domain to form a harmonious reasoning space. Experimental results across a broad range of domains show that MCPO alleviates performance degradation caused by mixed-domain training and outperforms single-domain training in most cases.

Primary area:
reinforcement learning
Keywords:
multi-domain post-training, contrastive learning, Large Reasoning Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uavpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Multi-domain Contrastive Policy Optimization (MCPO), a reinforcement-learning post-training framework designed for large reasoning models trained across heterogeneous domains simultaneously. The core idea is that cross-domain interactions are not inherently detrimental: certain interactions are supportive and can promote knowledge transfer, whereas others are competitive and help retain valuable distinctions among different reasoning patterns. MCPO begins by dividing each rollout into entropy-based reasoning segments, then estimates signed segment-level interactions through gradient inner products combined with a compact output-gradient proxy, and finally forms positive and negative segment pairs that are mutually augmented. Building on this, the framework proposes two multi-positive contrastive objectives—one for sharing knowledge across domains and the other for consolidating knowledge within a domain—which are optimized jointly alongside GRPO or DAPO. Experiments conducted on Math, Code, Science, ToolUse, and Safety demonstrate performance gains over multiple baselines. In addition, the paper presents extensive ablation studies covering the two contrastive scopes, segmentation granularity, relation retention, positive versus negative supervision, source-bank freshness, and backbone generalization.

Critical strengths

1.This paper addresses a significant challenge in multi-domain RL post-training: how a single shared policy can effectively learn from heterogeneous domains whose gradients may either conflict with or complement one another.

2.Rather than uniformly suppressing all cross-domain interference, the paper takes a more fine-grained view by distinguishing cross-domain interactions as either supportive or competitive.

3.The evaluation is comprehensive, covering five domains, nine benchmarks, two primary Qwen3 backbones, as well as additional DeepSeek-R1-Distill-Qwen and Llama backbones.

Critical weaknesses

1.The reported results are too weak to convincingly demonstrate the method's effectiveness. On Qwen-4B, the model achieves only 56.24/44.63 on AIME24/25 and 28.95/32.57 on LCB v5/v6. Is Qwen-4B-Base used as the initial model? If so, why not perform SFT first? Moreover, the baseline results are quite low—once the baselines are made sufficiently strong, the gains attributed to the method may shrink substantially or even vanish entirely.

2.The key assumption underlying the segmentation is not independently validated. MCPO assumes that high-entropy token positions mark the boundaries of reasoning structure. However, the evidence presented—that the selected cutpoints exhibit high entropy—is partly circular, since the cutpoints are chosen precisely by an entropy quantile rule. A high cutpoint-to-mean entropy ratio merely confirms that the procedure picks high-entropy locations; it does not establish that these locations correspond to meaningful semantic reasoning boundaries, nor that they constitute the appropriate units for cross-domain transfer.

3.The sparse gradient proxy is not empirically calibrated against full gradients or against actual loss changes over finite optimization steps.

4.The Euclidean local theory does not fully match practical Adam, clipping, and DAPO optimization.

Questions for authors

1.Which specific models do you use for initialization? Qwen3-4B-Base or Qwen3-4B? Qwen3-8B-Base or Qwen3-8B?

2.Why not perform SFT before RL? It is standard practice and doing so yields a stronger baseline, thereby providing a more reliable way to verify the method's actual effectiveness.

3.What are the proxy’s sign agreement and rank correlation with full gradients and finite-step updates?

4.How do the authors independently validate that entropy cutpoints represent genuine reasoning boundaries?

1 comment
AuthorsOpenReview ↗

Response to Reviewer uavp

Response to Question 1. Which specific models do you use for initialization? Qwen3-4B-Base or Qwen3-4B? Qwen3-8B-Base or Qwen3-8B?

As we noted in the experimental setup section, we explicitly use Qwen3-4B and Qwen3-8B, and specifies an 8,192-token response limit (Table 4, p. 25). Output budgets matter for this comparison: the Qwen3 report allows up to 38,912 tokens on AIME, while PPPO reports similar Qwen3-4B DAPO scores under a closer 10,240-token limit. We ask that these stated settings be considered when assessing the scores.

Response to Question 2. Why not perform SFT before RL? It is standard practice and doing so yields a stronger baseline, thereby providing a more reliable way to verify the method's actual effectiveness.

We focus our research on the effectiveness and robustness of multi-domain post-training methods. Meanwhile, fairness must be considered in controlled experiments against baselines. None of our compared approaches, such as MT-GRPO, MGS, CGPO, and various multimodal reinforcement learning methods, explicitly perform SFT in advance. Although SFT can boost performance, it raises another concern: does the performance gain stem from strong SFT or from different RT methods?

Response to Question 3. What are the proxy’s sign agreement and rank correlation with full gradients and finite-step updates?

Thank you very much for this supplementary question. It is a very valuable one, and we will prioritize validating them in future work.

Response to Question 4. How do the authors independently validate that entropy cutpoints represent genuine reasoning boundaries?

We adopt the entropy cutpoints method, which has been validated in papers such as SHAPE [1] and SPO [2].

[1] Ai, Zhengyang, et al. "Shape: Stage-aware hierarchical advantage via potential estimation for llm reasoning." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

[2] Guo, Yiran, et al. "Segment policy optimization: Effective segment-level credit assignment in rl for large language models." Advances in Neural Information Processing Systems 38 (2026): 114399-114431.

Reviewer_P1GYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-domain RL post-training and argues that cross-domain interactions should not be treated uniformly as interference: some reasoning structures may be supportive, while others are competing. It proposes MCPO, which segments rollouts into reasoning segments, estimates signed interactions between segments using gradient-based proxies, and introduces cross-domain and intra-domain contrastive objectives to encourage knowledge sharing. The method is generally well explained; unfortunately, the empirical results show only marginal gains.

Critical strengths
  1. The core motivation is intuitive.

  2. The implementation and experimental setup are documented thoroughly.

  3. The empirical coverage is reasonably broad in terms of tasks and baselines, spanning five domains and nine benchmarks, and comparing against six baselines.

Critical weaknesses
  1. The claimed cross-domain knowledge-sharing benefit is empirically weak. The most relevant comparison is against single-domain training, where MCPO-DAPO improves the overall score by only 0.35 points (55.85 vs. 55.50), with <1-point gains on Math and Code and degradation on ToolUse and Safety. This suggests that MCPO primarily recovers performance lost under mixed-domain training rather than demonstrating substantial beneficial transfer across domains. Given the considerable additional machinery of reasoning segmentation, gradient-based relation construction, contrastive objectives, and a source bank, I do not find the current magnitude of improvement sufficient to support the paper’s central claim.

  2. Following 1, given the very small reported gains and the apparent reliance on a single random seed, it is difficult to determine whether the improvements are attributable to the proposed method itself or simply to run-to-run variance.

  3. Following 1 and 2, the reported overall score averages nine benchmarks rather than macro-averaging the five domains, giving greater weight to Math, Code, and Science. Under an equal-weight domain macro-average, MCPO-DAPO achieves 58.32 compared with 58.19 for single-domain DAPO, reducing the claimed improvement from 0.35 to only 0.13 percentage points. Given such a marginal difference, especially without multi-seed uncertainty estimates, it is difficult to determine whether MCPO meaningfully outperforms single-domain training.

  4. The paper uses t-SNE visualizations to argue that MCPO creates a “harmonious reasoning space” with intra-domain compactness and cross-domain compatibility. However, there is no strong quantitative metric demonstrating that this geometry is systematically more compatible or predictive of transfer.

Questions for authors

How did you choose these domains? Please elaborate in greater details as they seem to differ from existing work [1].

[1] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2025).

1 comment
AuthorsOpenReview ↗

Response to Reviewer P1GY

Response to Question"How did you choose these domains? Please elaborate in greater details as they seem to differ from existing work [1]." [1] Cheng, Jorge Zhoujun, et al. "Revisiting reinforcement learning for llm reasoning from a cross-domain perspective." Advances in Neural Information Processing Systems 38 (2025).

We consider generality and practical value into account when selecting the domains. Multi-domain capabilities are critical for practically applicable Large Reasoning Models (LRMs). Specifically, Math, Code, and Science are widely recognized general reasoning domains. Tooluse serves as the foundation for agentic tasks. Safety is an indispensable consideration for the development of artificial intelligence. Moreover, their distinct distributions and pronounced domain heterogeneity further validate the robustness of our approach and support the general-purpose reasoners we introduced in the Introduction.

Reviewer_DgD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes MCPO, an RL post-training method that uses cross-domain reasoning interactions and contrastive objectives to improve multi-domain reasoning. It addresses an important problem with an interesting optimization perspective. However, concerns remain about the reliability of its interaction estimates, the justification for its contrastive objectives, and the strength of the empirical evidence supporting its main claims.

Critical strengths

S1. Important problem.

  • Addresses cross-domain interference in reasoning-model post-training, with an intuitive motivation for distinguishing supportive and competing interactions.

S2. Interesting framework.

  • Combines entropy-based segmentation, signed gradient interactions, and contrastive objectives for cross-domain sharing and intra-domain consolidation.

S3. Promising results.

  • MCPO-DAPO reports 55.85 on Qwen3-8B, outperforming DAPO (51.68) and CGPO (52.59), with ablations and additional backbone evaluations.

S4. Practicality.

  • Provides latency and memory profiling of MCPO's additional training overhead, helping assess implementation costs.
Critical weaknesses

W1. Proxy validity.

  • The gradient proxy's sign and ranking are not sufficiently validated against actual virtual source-step loss changes, leaving the central interaction-labeling mechanism uncertain.

W2. Mechanistic justification.

  • The link between local gradient interactions, contrastive objectives, and improved task reward or cross-domain generalization remains insufficiently established.

W3. Baseline and fairness.

  • MCPO-DAPO (55.85) is close to single-domain DAPO (55.50). The advantage of mixed-domain training needs clearer compute-budget and per-domain comparisons.

W4. Statistical reliability.

  • Results lack sufficiently clear multi-seed variability and uncertainty estimates to establish robustness.

W5. Indirect evidence.

  • t-SNE visualizations suggest knowledge sharing but do not demonstrate that representation geometry causes performance gains.

W6. Component attribution.

  • It remains unclear whether gains arise from the specific signed interaction mechanism or from generic contrastive regularization.
Questions for authors
  1. Can you validate the proxy’s sign and ranking against actual loss changes after virtual source-domain updates?

  2. How much does MCPO improve over single-domain DAPO when controlling for training and rollout budgets?

  3. Are the reported gains consistent across multiple random seeds, with confidence intervals or significance tests?

  4. Does MCPO improve performance on genuinely held-out domains, rather than only the evaluated training domains?

  5. Can you provide stronger evidence that the observed representation changes reflect cross-domain knowledge transfer rather than generic contrastive regularization?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Comment

Withdrawal Comment: Concerns About the Reviews of Submission 29560

We have decided to withdraw Submission 29560. We thank the reviewers and chairs for their time and would like to explain several concerns to our handling AC and the Program Chairs.

We suspect that the reviews by DgD5, P1GY, and uavp involved substantial AI assistance without sufficient checking against our manuscript. We recognize that the review text alone cannot establish how the reviews were produced. Our concern arises from recurring misunderstandings and questions about information already stated in the paper, as outlined below.

1. Distinguishing the Main Results from the Single-Domain Comparison

DgD5 and P1GY both request multiple random-seed runs and emphasize the small margin over single-domain DAPO. We agree that additional seeds would strengthen confidence in the results. P1GY describes the gains as “only marginal,” while DgD5 acknowledges the main-table results but also questions the mixed-domain advantage through the single-domain comparison.

Table 1 (p. 7) compares mixed-domain methods: MCPO improves over the strongest baseline by 3.13 points on Qwen3-4B and 3.26 points on Qwen3-8B. Table 2 (p. 9) examines an additional question: whether one mixed-domain model can match or outperform separately trained domain specialists. A small margin in this additional experiment does not make the main-table gains marginal. We are concerned that this distinction was not consistently reflected in the overall assessments.

2. The Intended Role of the Supplementary t-SNE Analysis

DgD5 asks whether representation geometry causes performance gains, while P1GY asks for quantitative evidence that the geometry predicts transfer. We agree that t-SNE alone cannot establish causality. Since contrastive learning is commonly studied in representation space, we include these plots to offer a supplementary interpretation of MCPO from that perspective (Figure 4 and Appendix D.2).

The benchmark comparisons and component ablations in Tables 1 and 2 provide the evidence for effectiveness. The plots help interpret the learned representations. Their limitations can constrain conclusions about representation geometry, but do not by themselves invalidate the performance results. We would appreciate a clearer distinction between these two types of evidence when evaluating the paper.

3. Model Initialization and the Shared Training Settings

uavp questions whether the initial models are Qwen3-4B/8B or their Base variants and describes the scores as unexpectedly weak. The paper names Qwen3-4B and Qwen3-8B (p. 7) and specifies an 8,192-token response limit (Table 4, p. 25). Output budgets matter for this comparison: the Qwen3 report allows up to 38,912 tokens on AIME, while PPPO reports similar Qwen3-4B DAPO scores under a closer 10,240-token limit. PPPO uses different training data, so it provides context rather than a matched comparison. We ask that these stated settings be considered when assessing the scores.

uavp also asks why supervised fine-tuning (SFT) was not performed before RL. MCPO and all reproduced baselines, including MT-GRPO and CGPO, start from the same released checkpoints without additional SFT. Appendices C.2–C.3 describe the shared settings. An additional SFT stage could be a useful extension if applied consistently across methods. However, whether it would reduce or eliminate MCPO’s gains remains an open experimental question.

Request to the AC and Program Chairs

We respectfully ask the AC and Program Chairs to examine these concerns and arrange a confidential check of the reviewers’ AI-use disclosures, original assessments, and relevant interaction records under the ICLR 2027 reviewer AI policy. If that examination confirms factual errors or policy violations, we ask that the official review record be corrected accordingly.

We acknowledge that the reviews also raise substantive questions, including proxy calibration and independent validation of entropy-based segmentation. We will consider these in further developing the work. Although we are withdrawing, we hope the concerns above receive attention so that the public record accurately reflects both the submission and the review process.

The Authors of Submission 29560

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.