- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work revisits the general reasoning ability of large language models from a geometric view. The authors find a mismatch between the policy manifold and the reward manifold during reinforcement learning. To solve this problem, they propose Decoupling and Coupling Reinforcement Learning, called DCRL.
DCRL has two key parts:
-
A prompt evolution mechanism based on syllogistic logic. It dynamically improves the reward rubric. This makes the reward manifold more expressive.
-
A policy–reward re-coupling mechanism. It updates the reward model and the policy model together. This keeps evaluation consistent. It also reduces manifold mismatch during training.
Critical strengths
The work finds that, in reward-based reinforcement learning, both the reward rubric and the reward model should not remain fixed. Instead, they should evolve dynamically during training. Based on this idea, the authors propose a corresponding algorithm, DCRL.
The paper is well written and well organized.
Critical weaknesses
Although the paper repeatedly mentions policy-reward manifold alignment, it does not truly analyze how manifold alignment affects model training. It also does not quantify the benefits of aligning the policy and reward models. As a result, this concept still feels more like a narrative framing for the paper. Figure 2 shows that models trained with DCRL have better policy-reward manifold alignment. However, this does not rigorously prove that the alignment is the cause of the performance improvement. When the reward model is trained jointly, its own capability may also improve. Therefore, the better training results may come from a stronger reward model or from other factors, rather than from policy-reward manifold alignment itself.
The theoretical foundations proposed in the paper are not rigorous enough and have low reliability. Many theoretical quantities are also difficult to compute in practice.
The experimental evaluation is insufficient. The experiments are conducted only on Qwen3-4B. This makes it difficult to show that the algorithm is generally applicable.
Questions for authors
Is policy-reward manifold alignment the reason why DCRL is effective? Does a reward model aligned with the policy model produce reward preferences that are consistent with the policy model, making it more likely to agree with the policy model? If the reward model is replaced with a stronger fixed model, would the reinforcement learning algorithm still be effective?
How is the prompt used for corollary prompt evolution in DCRL designed? What principles is it based on? Compared with prompt designs in other rubric self-evolution works, what advantages does it have?
Does DCRL remain effective on models other than Qwen3-4B? Compared with GRPO, PPO, and other algorithms, does the model trained with DCRL show any performance advantage?
In the actual training process, how do the corollary prompts change? Which changes make the reward more reliable?