acceptodds
Under review as a conference paper at ICLR 2027

Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving

Abstract

Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response's reward representation also depends on its randomly sampled group context, *i.e.*, the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose **Group-Marginalized Advantage Estimation (GMAE)**, which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Self-Rewarding Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KfUyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Self-rewarding RL methods suffer from high variability in reward estimation, considering that rewards for the same input may be different in different groups of samples. This submission aggregates rewards across multiple group contexts to estimate expected advantages to address this issue. Experiments show the proposed methods outperform existing methods, along with other analyses including training stability, generalization, and computational overhead.

Critical strengths

The motivation is interesting.

The experiments report decent improvement compared with existing self-rewarding methods.

Critical weaknesses

Too much overclaiming in this submission. One example is: Self-rewarding RL and self-evolution are quite different. This submission may contribute to self-rewarding but should not be overstated as self-evolution.

Poor reproducibility. This submission provides no code and few details about experimental configurations. The results may not be trustworthy given the current information.

Questions for authors

Q1. The submission includes some theoretical analysis about an approximation bound for the distribution of self-rewarding, but never aligns it with ground-truth correctness. Without such alignment, it is hard to tell if the algorithm theoretically converges, and converges to the optimal policy. How does GMAE address this fundamental limitation of self-rewarding?

Q2. The experiments report Pass@1, but not for larger sampling budgets (e.g., Pass@4, Pass@16, Pass@32, and Pass@128). Therefore, it is hard to tell if GMAE genuinely improves reasoning capability or only sharpens the output distribution.

By the way, I assigned a low score also because the submission contains some hallucinated bibliographic information and font-size issues, which may be considered for desk rejection.

2 comments

W1: overclaim about self-evolution.

We respectfully disagree with this concern. In the first paragraph of the Introduction, we use self-evolution as the broader research context and present self-rewarding RL as one approach to improving a policy through self-generated feedback on unlabeled prompts. We do not equate self-rewarding RL with all forms of self-evolution or claim it is necessary for them. We believe our framing does not constitute overclaiming.

W2: poor reproducibility.

We respectfully disagree that the reproducibility details are insufficient.

Hyperparameters. Section 5.1 provides the key training configurations, including the GRPO backbone, batch size (64), training epochs (1), learning rate (1×10−61\times10^{-6}), warmup ratio (5%), rollout size (16 main + 16 auxiliary responses), and sampling settings. We also specify the GPU hardware (NVIDIA A100 80GB). The Evaluation subsection further reports the inference sampling parameters (temperature 0.6, top-p 0.95, and top-k 20) and checkpoint selection protocol. If any details are still missing, we would appreciate it if you could point them out specifically.

Code Availability. We have included the anonymized code in the supplementary materials and will make the code publicly available upon acceptance of the paper.

W3: hallucinated bibliographic information and font-size issues

Hallucinated References. We strongly disagree with the allegation of hallucinated references. All bibliographic entries were manually retrieved from Google Scholar by us. We believe such a serious allegation should be supported by concrete evidence and would appreciate it if you could identify the specific problematic references.

Font Size. We have enlarged the fonts in Table 1 and Figure 2 in the revised manuscript to improve readability, while strictly adhering to the original 9-page submission limit.

Q1: The submission includes some theoretical analysis about an approximation bound for the distribution of self-rewarding, but never aligns it with ground-truth correctness. Without such alignment, it is hard to tell if the algorithm theoretically converges, and converges to the optimal policy. How does GMAE address this fundamental limitation of self-rewarding?

We can show that, under the same self-rewarding rule, adding GMAE yields a tighter training convergence bound with respect to ground-truth correctness.

Fix a self-rewarding rule (f,R)(f,R). For response yiy_i, let ci∈0,1c_i\in\\{0,1\\} be its ground-truth reward and ri(H)∈0,1r_i(H)\in\\{0,1\\} its self-reward under group context HH. The baseline uses one realized reward rir_i. GMAE instead constructs an empirical reward distribution P^i=K−1∑k=1Kδri(Hi(k))\hat P_i=K^{-1}\sum_{k=1}^{K}\delta_{r_i(H_i^{(k)})}, then computes each advantage as the expectation of the group-normalized advantage under ⨂jP^j\bigotimes_j\hat P_j. Writing p^i=P^i(1)\hat p_i=\hat P_i(\\{1\\}), the baseline corresponds to point masses at rir_i, GMAE to the distributions Bern⁡(p^i)\operatorname{Bern}(\hat p_i), and a ground-truth oracle to point masses at cic_i. They can therefore be compared through the same distribution-based advantage operator.

Let pi=E[ri(H)∣yi]p_i=\mathbb E[r_i(H)\mid y_i] and βi=∣pi−ci∣\beta_i=|p_i-c_i|. Because E[p^i∣yi]=pi\mathbb E[\hat p_i\mid y_i]=p_i, the reward representations satisfy

\mathbb E(\hat p_i-c_i)^2 =\beta_i^2+\operatorname{Var}(\hat p_i)\leq\beta_i.$$ The inequality follows from convexity: averaging rewards across contexts cannot increase their expected squared error to the fixed label $c_i$. To transfer this result to the actual GMAE advantage, we use its product-distribution expectation. Since a group-normalized advantage has magnitude at most $\sqrt{G-1}$, its difference from the ground-truth oracle advantage is bounded by $2\sqrt{G-1}\sum_j|\hat p_j-c_j|$. Under a bounded policy-score assumption, this yields upper bounds on the mean-squared update error relative to the *same* ground-truth oracle update: $$D_{\rm base}=C_G\sum_i\beta_i,\qquad D_{\rm GMAE} =C_G\sum_i\left[\beta_i^2+\operatorname{Var}(\hat p_i)\right],$$ where $C_G$ is shared. In particular, $$D_{\rm base}-D_{\rm GMAE} =C_G\sum_i\mathbb E[\hat p_i(1-\hat p_i)]\geq0.$$ The improvement is strict when sampled contexts give a response different rewards with positive probability. Now let $F_{\rm GT}$ be the ground-truth correctness objective. Assume it is $L$-smooth and bounded above, and that its oracle update has bias at most $B_0$ and rollout-noise standard deviation at most $\sigma_{\rm GT}$. Standard stochastic-ascent analysis then gives, for either method $m$ and $\eta\leq1/(4L)$, $$\frac1T\sum_{t=0}^{T-1}\mathbb E\\|\nabla F_{\rm GT}(\theta_t)\\|^2 \leq \frac{4\Delta_0}{\eta T} +3(B_0+\sqrt{D_m})^2 +2L\eta(\sigma_{\rm GT}+\sqrt{D_m})^2,$$ where $\Delta_0=F_{\rm GT}^{\max}-F_{\rm GT}(\theta_0)$. The ground-truth target and all other terms are shared in this matched comparison. $D_{\rm GMAE}\leq D_{\rm base}$ therefore makes GMAE’s convergence upper bound tighter. > **Q2:** The experiments report Pass@1, but not for larger sampling budgets (e.g., Pass@4, Pass@16, Pass@32, and Pass@128). Therefore, it is hard to tell if GMAE genuinely improves reasoning capability or only sharpens the output distribution. We supplement our evaluation with **Pass@16 results on Qwen3-4B-Base and Qwen3-8B-Base**, comparing the three GMAE variants against their corresponding baselines: TTRL, Co-Reward, and SR-TTRL. All evaluation settings remain consistent with the main experiments. The following table reports Pass@16 averaged across the eight benchmarks for each model. | Method | Qwen3-4B-Base | Qwen3-8B-Base | |---|---:|---:| | TTRL | 37.8 | 49.6 | | Co-Reward | 38.3 | 50.3 | | SR-TTRL | 37.1 | 48.7 | | $\mathrm{GMAE}_0$ | 39.2 | 51.6 | | $\mathrm{GMAE}_{\mathrm{CR}}$ | 39.9 | 52.0 | | $\mathrm{GMAE}_{\mathrm{SR}}$ | 40.1 | 51.6 | Compared with their corresponding baselines, $\mathrm{GMAE}_0$, $\mathrm{GMAE}_{\mathrm{CR}}$, and $\mathrm{GMAE}_{\mathrm{SR}}$ achieve **average Pass@16 improvements of** **1.7**, **1.7**, and **3.0 percentage points**, respectively, across the two models. These results provide further evidence that GMAE improves the likelihood of generating correct solutions.
Reviewer_R7jROpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Group-Marginalized Advantage Estimation (GMAE) for self-rewarding RL. Its key observation is that, in ensemble-based self-rewarding methods, a response's reward depends on randomly sampled peer responses in its rollout group. GMAE estimates a response-level reward distribution over alternative group contexts and computes an expected, normalized advantage. The approach is instantiated with voting, co-rewarding, and self-refinement rules, and evaluated across four base models and eight mathematics, science, knowledge, and coding benchmarks. The paper also studies calibration, sample-context budgets, training stability, datasets, and RL backbones.

Critical strengths
  1. A well-motivated and general intervention point. GMAE targets the dependence of a self-generated reward on the surrounding rollout group rather than committing to one particular pseudo-label construction. The three instantiations make the claimed compatibility with voting, co-rewarding, and self-refinement concrete.

  2. Strong breadth of evaluation. Table 1 spans four base models and eight benchmarks, while the later experiments test alternative datasets and two RL backbones. The five-seed controlled variance comparison (Figure 4) is particularly useful because stability is part of the stated motivation rather than an afterthought.

Critical weaknesses

1.Improvements could stem from context resampling, calibration, or an effective change in optimization scale rather than specifically from better marginalization of response-level reward uncertainty. Table 2 supports batch-level calibration, but a matched control that spends the same extra context/computation without forming the marginalized distribution would more directly isolate the proposed mechanism. Please add compute-matched resampling/averaging controls and report whether gains persist across reward functions with different context sensitivity. 2.The extremely small font sizes in Table 1 and Figure 2 are up to 76% smaller than the standard 10 pt text, which severely hinders readability.

Questions for authors

see weakness

3 comments

Approaching dealine of rebuttall, i do not get any response, so i change my rate to 2

Thank you for your constructive feedback. We address your concerns below.

W1: Table 2 supports batch-level calibration, but a matched control that spends the same extra context/computation without forming the marginalized distribution would more directly isolate the proposed mechanism. Please add compute-matched resampling/averaging controls and report whether gains persist across reward functions with different context sensitivity.

Thank you for suggesting this control. We introduce a compute-matched Resampling + Mean Reward (RMR) baseline and extend the comparison to all three GMAE variants.

1. Ablation design.

For a group of GG responses, RMR averages each response’s rewards over KK sampled contexts, rˉi=1K∑k=1Kri(k)\bar r_i=\frac{1}{K}\sum_{k=1}^{K}r_i^{(k)}, and then applies ordinary GRPO group normalization. GMAE instead computes the expectation of normalized advantages under the estimated reward distributions.

Formally, let Q=⨂j=1GBernoulli(rˉj)Q=\bigotimes_{j=1}^{G}\mathrm{Bernoulli}(\bar r_j) and define Ni(v)=(vi−μ(v))/σ(v)\mathcal N_i(\mathbf v)=(v_i-\mu(\mathbf v))/\sigma(\mathbf v), assigning zero advantages when σ(v)=0\sigma(\mathbf v)=0. The two mechanisms are

=\mathcal N_i\left(\mathbb E_{\mathbf v\sim Q}[\mathbf v]\right), \qquad A_i^{\mathrm{GMAE}} =\mathbb E_{\mathbf v\sim Q}\left[\mathcal N_i(\mathbf v)\right].$$ Thus, RMR **normalizes averaged rewards**, while GMAE **averages normalized advantages**. Each RMR control uses the same rollout/context budgets and reward-construction procedure. **2. Experimental comparison.** We evaluate on **Qwen3-8B-Base**, reporting average Mean@16 across all eight benchmarks. We pair **TTRL + RMR**, **Co-Reward + RMR**, and **SR-TTRL + RMR** with their corresponding GMAE variants. Original baseline and GMAE scores are reproduced from Table 1. | Base method | Base Avg. | Base + RMR Avg. | GMAE variant | GMAE Avg. | |---|---|---|---|---| | TTRL | 34.5 | 35.3 | $\mathrm{GMAE}_{\mathrm{0}}$ | **37.1** | | Co-Reward | 34.9 | 35.6 | $\mathrm{GMAE}_{\mathrm{CR}}$ | **38.0** | | SR-TTRL | 34.6 | 35.3 | $\mathrm{GMAE}_{\mathrm{SR}}$ | **36.9** | **RMR improves all three original baselines, while GMAE consistently outperforms the corresponding RMR controls across all three reward constructions.** These results support the benefit of marginalized advantage computation beyond additional context resampling and reward averaging. > **W2:** The extremely small font sizes in Table 1 and Figure 2 severely hinder readability. We apologize for the readability issues. We have increased the font sizes in Table 1 and Figure 2 while keeping the revised manuscript within the original nine-page limit.

As the first author, I sincerely apologize for the delay due to personal matters. We are working through all three reviews one by one. We hope you can consider our responses before making your final decision. Thank you very much for your understanding.

Reviewer_PHuWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets a practical limitation of ensemble‑based self‑rewarding RL for zero‑label LLM self‑evolution. Existing methods compute rewards from one random rollout group, so reward estimates are unstable and can flip sign depending on other sampled responses. The authors propose GMAE, group‑marginalized advantage estimation. It samples multiple alternative group contexts from a shared candidate pool to build an empirical reward distribution, and computes expected advantages over this distribution. A batch‑level calibration mitigates advantage scale reduction.

GMAE works as a wrapper compatible with multiple existing self‑reward rules, giving three variants. Experiments across four models and eight benchmarks show consistent accuracy gains, lower training variance and moderate computation overhead. The approach retains strong out‑of‑domain performance when only trained on mathematical data.

Critical strengths
  1. The paper identifies a real under‑discussed issue: group‑context randomness corrupts reward signals in ensemble self‑reward RL, with empirical evidence showing sign‑flipped advantages across contexts.
  2. GMAE is general and modular, it can plug into different self‑reward pipelines instead of being tied to one specific reward design. Theoretical analysis provides error bounds and motivates batch‑level calibration over per‑prompt calibration.
  3. Empirical evaluation is comprehensive. It covers multiple model families, in‑domain and out‑of‑domain benchmarks, multi‑seed stability tests, ablations for hyperparameters, plus validation across different RL backbones and training datasets.
  4. Extra computation mainly consists of lightweight scalar operations instead of expensive model forward passes, leading to reasonable overhead for real‑world training workflows.
Critical weaknesses
  1. Failure modes are not well explored. If most sampled group contexts produce wrong pseudo‑labels, marginalizing over these noisy contexts might average biased signals. Controlled experiments for this edge case are missing.
  2. Exact implementation enumerates 2^G joint reward outcomes. This creates scalability risks for large rollout group sizes commonly used in GRPO, and approximate alternatives are not evaluated.
  3. There is no empirical comparison against zero‑label LLM‑as‑judge self‑RL. Even a conceptual discussion would help position this work within broader self‑improvement literature.
  4. Strong out‑of‑domain generalization is observed but not deeply analyzed. It remains unclear why reduced in‑domain training noise brings consistent cross‑domain gains.
Questions for authors
  1. When most group contexts give wrong pseudo‑labels, will marginalization average away noise or reinforce bias? Have you tested small controlled cases for this scenario?
  2. Exact enumeration is costly for large rollout groups. What performance trade‑offs would approximate sampling bring?
  3. Why does lower training noise on math data lead to consistent gains on unseen science and coding benchmarks?
3 comments

Thanks for your constructive reviews. We address your concerns one by one as follows.

W1 + Q1: When most group contexts give wrong pseudo-labels, will marginalization average away noise or reinforce bias? Have you tested small controlled cases for this scenario?

Thank you for highlighting this case. We added a controlled voting simulation to examine how marginalization affects a correct response when most contexts produce incorrect pseudo-labels.

Each original group contains 16 responses, represented by an answer-count vector. The first entry counts responses giving the correct answer y0y_0, while a×ka^{\times k} denotes kk distinct incorrect answers appearing aa times each. For example, (4,6×1,1×6)(4,6^{\times1},1^{\times6}) contains four correct responses, one incorrect answer appearing six times, and six other incorrect answers appearing once each. We track one correct response and add 16 auxiliary responses with the same answer-count profile, forming a 32-response pool. Voting ties are broken uniformly among the tied winners.

For each original response, we uniformly select 15 of the other 31 responses as its context, compute exact reward marginals through combinatorial counting, and evaluate the expected advantage under the product distribution in Eq. 10. We compare the focal correct response’s advantage A0A_0 under the original group’s incorrect pseudo-label with its marginalized expectation E[A0]\mathbb E[A_0]. Context error denotes the proportion of contexts yielding an incorrect pseudo-label for the group containing this response.

Original group counts (16 responses)Answer classesContext errorOriginal A0A_0 under an incorrect pseudo-labelMarginalized E[A0]\mathbb E[A_0]
(5,7×1,1×4)(5,7^{\times1},1^{\times4})671.87%−0.8819−0.3552
(4,6×1,1×6)(4,6^{\times1},1^{\times6})873.77%−0.7746−0.2679
(2,3×1,2×3,1×5)(2,3^{\times1},2^{\times3},1^{\times5})1078.51%−0.4804−0.0267
(3,4×1,3×1,2×3)(3,4^{\times1},3^{\times1},2^{\times3})671.63%−0.5774−0.0130
(2,3×1,2×2,1×7)(2,3^{\times1},2^{\times2},1^{\times7})1175.45%−0.48040.0542
(2,3×1,2×1,1×9)(2,3^{\times1},2^{\times1},1^{\times9})1271.67%−0.48040.1559
(3,4×1,3×1,2×2,1×2)(3,4^{\times1},3^{\times1},2^{\times2},1^{\times2})770.83%−0.57740.0057
(4,5×1,3×1,2×2)(4,5^{\times1},3^{\times1},2^{\times2})562.59%−0.67420.0580

Across all eight controlled settings, marginalization reduces the erroneous suppression of the correct response, with E[A0]>A0\mathbb E[A_0]>A_0 in every case. It can even reverse the advantage from negative to positive, including when more than 70% of contexts produce incorrect pseudo-labels. These results demonstrate that marginalization can recover useful learning signals despite predominantly incorrect contexts.

By preserving correct evidence across contexts, GMAE can reduce premature suppression of correct responses when the answer distribution remains diffuse early in training, consistent with Fig. 6.

W2 + Q2: Exact implementation enumerates 2G2^G joint reward outcomes. This creates scalability risks for large rollout groups. What performance trade-offs would approximate sampling bring?

Thank you for raising the scalability concern. For a group of GG responses, let P^rj\widehat P_{r_j} denote the empirical reward distribution of response jj. Since our rewards are binary, this distribution is fully specified by pjp_j, its probability of receiving reward 1. Each v=(v1,…,vG)∈{0,1}G\mathbf v=(v_1,\ldots,v_G)\in\{0,1\}^G represents one possible joint reward realization. Eq. 10 can therefore be written as

AiGMAE=Ev∼⨂j=1GP^rj[vi−μvσv]=∑v∈{0,1}G[∏j=1Gpjvj(1−pj)1−vj]vi−μvσv.A_i^{\mathrm{GMAE}}=\mathbb E_{\mathbf v\sim\bigotimes_{j=1}^{G}\widehat P_{r_j}}\left[\frac{v_i-\mu_{\mathbf v}}{\sigma_{\mathbf v}}\right]=\sum_{\mathbf v\in\{0,1\}^{G}}\left[\prod_{j=1}^{G}p_j^{v_j}(1-p_j)^{1-v_j}\right]\frac{v_i-\mu_{\mathbf v}}{\sigma_{\mathbf v}}.

The normalized advantage is defined as zero when σv=0\sigma_{\mathbf v}=0. Direct evaluation enumerates 2G2^G reward vectors. However, all GG expected advantages can be computed exactly in O(G2)\mathcal O(G^2) time using dynamic programming. For a given reward vector, let s=∑j=1Gvjs=\sum_{j=1}^{G}v_j. Binary rewards give μv=s/G\mu_{\mathbf v}=s/G and σv=s(G−s)/G\sigma_{\mathbf v}=\sqrt{s(G-s)}/G. Define h(s)=1/s(G−s)h(s)=1/\sqrt{s(G-s)} for 0<s<G0<s<G, with h(0)=h(G)=0h(0)=h(G)=0. Then

=\mathbb E_{\mathbf v}\left[(Gv_i-s)h(s)\right] =Gu_i-\sum_{j=1}^{G}u_j, \qquad u_i=\mathbb E_{\mathbf v}\left[v_i h(s)\right].$$ To compute all $u_i$, define the prefix count probabilities $F_i(s)=\Pr(\sum_{j=1}^{i}v_j=s)$ and suffix expectations $B_i(t)=\mathbb E[h(t+\sum_{j=i+1}^{G}v_j)]$. Starting from $F_0(0)=1$ and $B_G(t)=h(t)$, with out-of-range prefix probabilities set to zero, we use $$\begin{aligned} F_i(s)&=(1-p_i)F_{i-1}(s)+p_iF_{i-1}(s-1), &&s=0,\ldots,i,\\\\ u_i&=p_i\sum_{s=0}^{i-1}F_{i-1}(s)B_i(s+1),\\\\ B_{i-1}(t)&=(1-p_i)B_i(t)+p_iB_i(t+1), &&t=0,\ldots,i-1. \end{aligned}$$ The forward pass stores the prefix probabilities. The backward pass proceeds through $i=G,\ldots,1$, computing $u_i$ before updating $B_{i-1}$. The expression for $u_i$ follows by conditioning on $v_i=1$: a prefix with $s$ positive rewards contributes $s+1$ after including response $i$. Each position requires $\mathcal O(i)$ operations, yielding **$\mathcal O(G^2)$ total time for all advantages**. Storing the prefix probabilities requires $\mathcal O(G^2)$ space, while the backward array uses $\mathcal O(G)$ additional space. Thus, given the empirical reward marginals, Eq. 10 can be evaluated exactly at quadratic cost. **Approximate joint-reward sampling is therefore unnecessary to avoid exponential enumeration in our binary-reward setting.**

W3: There is no empirical comparison against zero-label LLM-as-judge self-RL. Even a conceptual discussion would help position this work within broader self-improvement literature.

Thank you for highlighting this setting. RLSR [1] studies self-judging, where the actor also serves as the judge to generate rewards without reference answers. We compare GMAE₀ with a self-judge baseline using Qwen3-4B and Qwen3-8B as actors. We match the training data and GRPO update budget and report average accuracy across all eight benchmarks. Total training cost includes auxiliary rollouts and judging and is normalized to GMAE₀ separately for each actor.

MethodQwen3-4B Avg. (%)Relative cost (4B)Qwen3-8B Avg. (%)Relative cost (8B)
Self-judge25.51.65×34.41.71×
GMAE₀28.91.00×37.11.00×

GMAE₀ outperforms self-judging on both actors at lower training cost. We will include this comparison and the conceptual discussion in the revised appendix.

W4 + Q3: Strong out-of-domain generalization is observed but not deeply analyzed. Why does lower training noise on math data lead to consistent gains on unseen science and coding benchmarks?

Thank you for highlighting this point. First, cross-domain gains have also been observed in other reasoning RL methods [2][3]. Tasks across domains can share reasoning behaviors, such as verification and error correction. Strengthening these behaviors during mathematical training can therefore benefit other tasks that rely on them.

Second, within self-rewarding RL, GMAE further improves cross-domain performance over its corresponding baselines (Table 1). We interpret this additional gain through the better preservation of useful reasoning behaviors. Noisy self-rewards can penalize correct but infrequently sampled solutions. By retaining positive evidence across group contexts, GMAE reduces their premature suppression, preserving opportunities to reinforce reasoning behaviors that may transfer across tasks.

This interpretation is consistent with Fig. 6, where more prompts retain occasional correct responses under GMAE. These preserved trajectories provide further opportunities for learning, helping explain the additional OOD gains.


References

[1] RLSR: Reinforcement Learning from Self Reward.

[2] Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective.

[3] S²R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.