- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Multi-domain Contrastive Policy Optimization (MCPO), a reinforcement-learning post-training framework designed for large reasoning models trained across heterogeneous domains simultaneously. The core idea is that cross-domain interactions are not inherently detrimental: certain interactions are supportive and can promote knowledge transfer, whereas others are competitive and help retain valuable distinctions among different reasoning patterns. MCPO begins by dividing each rollout into entropy-based reasoning segments, then estimates signed segment-level interactions through gradient inner products combined with a compact output-gradient proxy, and finally forms positive and negative segment pairs that are mutually augmented. Building on this, the framework proposes two multi-positive contrastive objectives—one for sharing knowledge across domains and the other for consolidating knowledge within a domain—which are optimized jointly alongside GRPO or DAPO. Experiments conducted on Math, Code, Science, ToolUse, and Safety demonstrate performance gains over multiple baselines. In addition, the paper presents extensive ablation studies covering the two contrastive scopes, segmentation granularity, relation retention, positive versus negative supervision, source-bank freshness, and backbone generalization.
Critical strengths
1.This paper addresses a significant challenge in multi-domain RL post-training: how a single shared policy can effectively learn from heterogeneous domains whose gradients may either conflict with or complement one another.
2.Rather than uniformly suppressing all cross-domain interference, the paper takes a more fine-grained view by distinguishing cross-domain interactions as either supportive or competitive.
3.The evaluation is comprehensive, covering five domains, nine benchmarks, two primary Qwen3 backbones, as well as additional DeepSeek-R1-Distill-Qwen and Llama backbones.
Critical weaknesses
1.The reported results are too weak to convincingly demonstrate the method's effectiveness. On Qwen-4B, the model achieves only 56.24/44.63 on AIME24/25 and 28.95/32.57 on LCB v5/v6. Is Qwen-4B-Base used as the initial model? If so, why not perform SFT first? Moreover, the baseline results are quite low—once the baselines are made sufficiently strong, the gains attributed to the method may shrink substantially or even vanish entirely.
2.The key assumption underlying the segmentation is not independently validated. MCPO assumes that high-entropy token positions mark the boundaries of reasoning structure. However, the evidence presented—that the selected cutpoints exhibit high entropy—is partly circular, since the cutpoints are chosen precisely by an entropy quantile rule. A high cutpoint-to-mean entropy ratio merely confirms that the procedure picks high-entropy locations; it does not establish that these locations correspond to meaningful semantic reasoning boundaries, nor that they constitute the appropriate units for cross-domain transfer.
3.The sparse gradient proxy is not empirically calibrated against full gradients or against actual loss changes over finite optimization steps.
4.The Euclidean local theory does not fully match practical Adam, clipping, and DAPO optimization.
Questions for authors
1.Which specific models do you use for initialization? Qwen3-4B-Base or Qwen3-4B? Qwen3-8B-Base or Qwen3-8B?
2.Why not perform SFT before RL? It is standard practice and doing so yields a stronger baseline, thereby providing a more reliable way to verify the method's actual effectiveness.
3.What are the proxy’s sign agreement and rank correlation with full gradients and finite-step updates?
4.How do the authors independently validate that entropy cutpoints represent genuine reasoning boundaries?
Response to Reviewer uavp
Response to Question 1. Which specific models do you use for initialization? Qwen3-4B-Base or Qwen3-4B? Qwen3-8B-Base or Qwen3-8B?
As we noted in the experimental setup section, we explicitly use Qwen3-4B and Qwen3-8B, and specifies an 8,192-token response limit (Table 4, p. 25). Output budgets matter for this comparison: the Qwen3 report allows up to 38,912 tokens on AIME, while PPPO reports similar Qwen3-4B DAPO scores under a closer 10,240-token limit. We ask that these stated settings be considered when assessing the scores.
Response to Question 2. Why not perform SFT before RL? It is standard practice and doing so yields a stronger baseline, thereby providing a more reliable way to verify the method's actual effectiveness.
We focus our research on the effectiveness and robustness of multi-domain post-training methods. Meanwhile, fairness must be considered in controlled experiments against baselines. None of our compared approaches, such as MT-GRPO, MGS, CGPO, and various multimodal reinforcement learning methods, explicitly perform SFT in advance. Although SFT can boost performance, it raises another concern: does the performance gain stem from strong SFT or from different RT methods?
Response to Question 3. What are the proxy’s sign agreement and rank correlation with full gradients and finite-step updates?
Thank you very much for this supplementary question. It is a very valuable one, and we will prioritize validating them in future work.
Response to Question 4. How do the authors independently validate that entropy cutpoints represent genuine reasoning boundaries?
We adopt the entropy cutpoints method, which has been validated in papers such as SHAPE [1] and SPO [2].
[1] Ai, Zhengyang, et al. "Shape: Stage-aware hierarchical advantage via potential estimation for llm reasoning." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
[2] Guo, Yiran, et al. "Segment policy optimization: Effective segment-level credit assignment in rl for large language models." Advances in Neural Information Processing Systems 38 (2026): 114399-114431.