acceptodds
Withdrawn from ICLR 2027

DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) has emerged as a key paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing reward systems, such as rule-based and reward-model-based, often exhibit issues such as unstable optimization and reward hacking. In this work, we revisit the general reasoning of LLMs from a geometric perspective, conceptualizing it as a coupled manifold composed of three interdependent sub-manifolds: logical deduction, evaluation, and representation. Based on this perspective, response generation in RL can be interpreted as a decoupling process from the evaluation manifold, while reward estimation corresponds to a decoupling process from the logical deduction manifold. The limitations of rule-based and reward-model RL systems can be geometrically interpreted as the mismatch of policy-reward manifolds during RL process. To address the aforementioned misalignment, we propose Decoupling and Coupling Reinforcement Learning (DCRL) framework, which incorporates two key components: (1) a syllogistic logic-based prompt evolution mechanism that dynamically refines reward rubrics to enhance the expressiveness of the reward manifold; and (2) a policy–reward re-coupling mechanism that jointly updates the reward and policy models, ensuring consistent evaluation and mitigating manifold mismatch during training. Theoretical analysis and extensive experiments across multiple reasoning domains demonstrate that DCRL consistently outperforms both rule-based and reward-model baselines. Notably, a Qwen3-4B model trained under DCRL surpasses a Qwen3-32B baseline and approaches the performance of a Qwen3-235B model, highlighting superior effectiveness and generalization in RL.

Primary area:
reinforcement learning
Keywords:
LLM reasoning, manifold theory, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_tpANOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work revisits the general reasoning ability of large language models from a geometric view. The authors find a mismatch between the policy manifold and the reward manifold during reinforcement learning. To solve this problem, they propose Decoupling and Coupling Reinforcement Learning, called DCRL.

DCRL has two key parts:

  • A prompt evolution mechanism based on syllogistic logic. It dynamically improves the reward rubric. This makes the reward manifold more expressive.

  • A policy–reward re-coupling mechanism. It updates the reward model and the policy model together. This keeps evaluation consistent. It also reduces manifold mismatch during training.

Critical strengths

The work finds that, in reward-based reinforcement learning, both the reward rubric and the reward model should not remain fixed. Instead, they should evolve dynamically during training. Based on this idea, the authors propose a corresponding algorithm, DCRL.

The paper is well written and well organized.

Critical weaknesses

Although the paper repeatedly mentions policy-reward manifold alignment, it does not truly analyze how manifold alignment affects model training. It also does not quantify the benefits of aligning the policy and reward models. As a result, this concept still feels more like a narrative framing for the paper. Figure 2 shows that models trained with DCRL have better policy-reward manifold alignment. However, this does not rigorously prove that the alignment is the cause of the performance improvement. When the reward model is trained jointly, its own capability may also improve. Therefore, the better training results may come from a stronger reward model or from other factors, rather than from policy-reward manifold alignment itself.

The theoretical foundations proposed in the paper are not rigorous enough and have low reliability. Many theoretical quantities are also difficult to compute in practice.

The experimental evaluation is insufficient. The experiments are conducted only on Qwen3-4B. This makes it difficult to show that the algorithm is generally applicable.

Questions for authors

Is policy-reward manifold alignment the reason why DCRL is effective? Does a reward model aligned with the policy model produce reward preferences that are consistent with the policy model, making it more likely to agree with the policy model? If the reward model is replaced with a stronger fixed model, would the reinforcement learning algorithm still be effective?

How is the prompt used for corollary prompt evolution in DCRL designed? What principles is it based on? Compared with prompt designs in other rubric self-evolution works, what advantages does it have?

Does DCRL remain effective on models other than Qwen3-4B? Compared with GRPO, PPO, and other algorithms, does the model trained with DCRL show any performance advantage?

In the actual training process, how do the corollary prompts change? Which changes make the reward more reliable?

Reviewer_yYBiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes DCRL, an RL framework that combines a dynamically evolving reward rubric with periodic synchronization of the reward model to the updated policy. The rubric is organized into fixed axioms, domain-level theorems, and evolving corollaries. The authors motivate this design through a geometric account of reasoning, evaluation, and representation manifolds, and claim that synchronization and prompt evolution keep the policy–reward reasoning gap bounded. Experiments train a Qwen3-4B policy on mathematical, coding, and general reasoning tasks and report improvements over rule-based and static reward-model alternatives.

Critical strengths
  1. The paper addresses an important practical problem: reward quality can deteriorate as the policy's response distribution changes. Updating evaluation criteria alongside policy training is a worthwhile direction.
  2. The proposed implementation is conceptually simple. Hierarchical rubric updates and periodic weight synchronization could be incorporated into existing RL pipelines without introducing a separately supervised reward-model training stage.
  3. The experiments cover several task families and include out-of-distribution evaluations. Some reported improvements are substantial; for example, Table 2 reports gains over the static-reward baseline on CodeContest and GPQA.
Critical weaknesses

W1. The main theoretical guarantee relies on an unjustified identification of conditional distributions. In Appendix A.2, Eq. (30) concludes that the reward and policy reasoning distributions are identical immediately after their parameters are synchronized. Equal parameters do not imply equal distributions when the inputs and tasks differ: a policy generates a solution, whereas the reward model evaluates a response under a rubric. Moreover, Assumption 3.1 controls the difference between reward distributions under two nearby rubric prompts. It does not establish the comparison between the policy distribution and the reward distribution used in Eq. (34). An additional condition identifying the policy distribution with the reward distribution under the previous rubric is needed. The proof also does not bound the drift caused by policy updates between synchronization events. Consequently, Theorem 3.3 does not currently establish the claimed uniform alignment during training. Even a valid distributional proximity bound would require an additional argument to imply reward accuracy or resistance to reward hacking.

W2. The geometric interpretation is not sufficiently connected to measurable evaluation quality. The introduction uses unions of manifolds and orthogonality to characterize reasoning modes, but does not specify an operational construction that validates these objects or the claimed decoupling. More importantly, Appendix B.2 measures alignment using TF-IDF followed by SVD, or a hashed bag-of-tokens fallback, applied to generated solution and judge text. These are observable text-space proxies, as the appendix acknowledges. They could reflect vocabulary, formatting, or the evaluator's reuse of response content. They do not directly establish alignment of latent reasoning manifolds or accurate judgments. A more convincing validation would measure whether the judge better distinguishes correct from incorrect responses on fresh policy outputs, using independent task verifiers.

W3. The ablations do not isolate rubric evolution from parameter synchronization. Figure 3 compares the complete method with static-model and rule-based alternatives, but does not provide a clear factorial comparison of a fixed rubric versus an evolving rubric, crossed with a frozen judge versus periodic policy-to-judge copying. This matters because either a better prompt or a changing judge could explain the improvements without the proposed manifold interpretation. Synchronizing a judge with a policy optimized for solving tasks also does not automatically improve its evaluation ability. Direct measurements of judge accuracy before and after synchronization are needed to test this assumption.

W4. Important reward and evaluation details remain ambiguous. Algorithm 1 describes a rubric-derived reward, whereas Appendix C states that execution-based evaluation is the primary signal and the judge score is auxiliary. The exact combined reward, its scaling, and its use across domains should be specified. The paper should also document the actual optimizer configuration, rubric-update frequency and acceptance procedure, sampling protocol for pass@5, and repeated-run variability. The stated 9:1 partitioning rule does not straightforwardly explain every train/test count in Table 1. Finally, the comparison with much larger zero-shot models should be described as task-specific post-training versus zero-shot reference performance, rather than as a general replacement for model scale.

Questions for authors

What additional assumptions justify Eq. (30) and the application of prompt smoothness in Eq. (34), given the different policy and evaluator conditioning contexts? Can the authors report the four combinations of fixed/evolving rubrics and frozen/synchronized judges, together with independently measured judge accuracy? What is the exact scalar reward used for each domain, including any verifier component and its coefficient? If the advantages are group-centered as in GRPO, what useful information does their mean provide to the rubric-update rule? How are rubric candidates selected, and what are the train/test partitions, pass@5 estimator, inference settings, and additional compute costs?

Reviewer_JbQtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DCRL, a reinforcement learning framework combining hierarchical reward-rubric evolution with periodic synchronization of reward-model parameters to the updated policy. The rubric contains fixed axioms, domain-specific rules, and dynamically updated heuristics. The authors motivate the design through policy–reward manifold alignment, provide a claimed uniform divergence bound, and report improvements across mathematical, coding, and general reasoning benchmarks using Qwen3-4B.

Critical strengths
  • The paper addresses an important problem: maintaining reliable reward supervision as the policy changes during reinforcement learning. The combination of adaptive rubrics and parameter synchronization is straightforward to understand.
  • The evaluation covers multiple reasoning domains and includes out-of-distribution tests. Table 2 reports consistent improvements over the rule-based and static-reward baselines.
Critical weaknesses
  • Eq. (30) assumes that shared weights imply identical reasoning distributions, despite generation and evaluation using different inputs and instructions. Eq. (34) then applies prompt smoothness to the policy–reward gap without establishing the necessary connection.

  • Figure 3 does not separate rubric evolution from parameter synchronization, and text-based alignment scores do not establish better reward accuracy. Appendix C also describes execution rewards as primary and model scores as auxiliary, but leaves their combination unspecified.

  • DCRL combines rubric rewriting with policy-to-reward weight synchronization, but the paper does not clearly establish what this adds beyond existing self-rewarding and rubric-based methods. The manifold framing provides an interpretation rather than a distinct optimization procedure, and the experiments lack direct comparisons that would demonstrate the value of the proposed combination.

Questions for authors

See Weaknesses, please.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.