- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Self-rewarding RL methods suffer from high variability in reward estimation, considering that rewards for the same input may be different in different groups of samples. This submission aggregates rewards across multiple group contexts to estimate expected advantages to address this issue. Experiments show the proposed methods outperform existing methods, along with other analyses including training stability, generalization, and computational overhead.
Critical strengths
The motivation is interesting.
The experiments report decent improvement compared with existing self-rewarding methods.
Critical weaknesses
Too much overclaiming in this submission. One example is: Self-rewarding RL and self-evolution are quite different. This submission may contribute to self-rewarding but should not be overstated as self-evolution.
Poor reproducibility. This submission provides no code and few details about experimental configurations. The results may not be trustworthy given the current information.
Questions for authors
Q1. The submission includes some theoretical analysis about an approximation bound for the distribution of self-rewarding, but never aligns it with ground-truth correctness. Without such alignment, it is hard to tell if the algorithm theoretically converges, and converges to the optimal policy. How does GMAE address this fundamental limitation of self-rewarding?
Q2. The experiments report Pass@1, but not for larger sampling budgets (e.g., Pass@4, Pass@16, Pass@32, and Pass@128). Therefore, it is hard to tell if GMAE genuinely improves reasoning capability or only sharpens the output distribution.
By the way, I assigned a low score also because the submission contains some hallucinated bibliographic information and font-size issues, which may be considered for desk rejection.
2 comments
Q1: The submission includes some theoretical analysis about an approximation bound for the distribution of self-rewarding, but never aligns it with ground-truth correctness. Without such alignment, it is hard to tell if the algorithm theoretically converges, and converges to the optimal policy. How does GMAE address this fundamental limitation of self-rewarding?
We can show that, under the same self-rewarding rule, adding GMAE yields a tighter training convergence bound with respect to ground-truth correctness.
Fix a self-rewarding rule . For response , let be its ground-truth reward and its self-reward under group context . The baseline uses one realized reward . GMAE instead constructs an empirical reward distribution , then computes each advantage as the expectation of the group-normalized advantage under . Writing , the baseline corresponds to point masses at , GMAE to the distributions , and a ground-truth oracle to point masses at . They can therefore be compared through the same distribution-based advantage operator.
Let and . Because , the reward representations satisfy
\mathbb E(\hat p_i-c_i)^2 =\beta_i^2+\operatorname{Var}(\hat p_i)\leq\beta_i.$$ The inequality follows from convexity: averaging rewards across contexts cannot increase their expected squared error to the fixed label $c_i$. To transfer this result to the actual GMAE advantage, we use its product-distribution expectation. Since a group-normalized advantage has magnitude at most $\sqrt{G-1}$, its difference from the ground-truth oracle advantage is bounded by $2\sqrt{G-1}\sum_j|\hat p_j-c_j|$. Under a bounded policy-score assumption, this yields upper bounds on the mean-squared update error relative to the *same* ground-truth oracle update: $$D_{\rm base}=C_G\sum_i\beta_i,\qquad D_{\rm GMAE} =C_G\sum_i\left[\beta_i^2+\operatorname{Var}(\hat p_i)\right],$$ where $C_G$ is shared. In particular, $$D_{\rm base}-D_{\rm GMAE} =C_G\sum_i\mathbb E[\hat p_i(1-\hat p_i)]\geq0.$$ The improvement is strict when sampled contexts give a response different rewards with positive probability. Now let $F_{\rm GT}$ be the ground-truth correctness objective. Assume it is $L$-smooth and bounded above, and that its oracle update has bias at most $B_0$ and rollout-noise standard deviation at most $\sigma_{\rm GT}$. Standard stochastic-ascent analysis then gives, for either method $m$ and $\eta\leq1/(4L)$, $$\frac1T\sum_{t=0}^{T-1}\mathbb E\\|\nabla F_{\rm GT}(\theta_t)\\|^2 \leq \frac{4\Delta_0}{\eta T} +3(B_0+\sqrt{D_m})^2 +2L\eta(\sigma_{\rm GT}+\sqrt{D_m})^2,$$ where $\Delta_0=F_{\rm GT}^{\max}-F_{\rm GT}(\theta_0)$. The ground-truth target and all other terms are shared in this matched comparison. $D_{\rm GMAE}\leq D_{\rm base}$ therefore makes GMAE’s convergence upper bound tighter. > **Q2:** The experiments report Pass@1, but not for larger sampling budgets (e.g., Pass@4, Pass@16, Pass@32, and Pass@128). Therefore, it is hard to tell if GMAE genuinely improves reasoning capability or only sharpens the output distribution. We supplement our evaluation with **Pass@16 results on Qwen3-4B-Base and Qwen3-8B-Base**, comparing the three GMAE variants against their corresponding baselines: TTRL, Co-Reward, and SR-TTRL. All evaluation settings remain consistent with the main experiments. The following table reports Pass@16 averaged across the eight benchmarks for each model. | Method | Qwen3-4B-Base | Qwen3-8B-Base | |---|---:|---:| | TTRL | 37.8 | 49.6 | | Co-Reward | 38.3 | 50.3 | | SR-TTRL | 37.1 | 48.7 | | $\mathrm{GMAE}_0$ | 39.2 | 51.6 | | $\mathrm{GMAE}_{\mathrm{CR}}$ | 39.9 | 52.0 | | $\mathrm{GMAE}_{\mathrm{SR}}$ | 40.1 | 51.6 | Compared with their corresponding baselines, $\mathrm{GMAE}_0$, $\mathrm{GMAE}_{\mathrm{CR}}$, and $\mathrm{GMAE}_{\mathrm{SR}}$ achieve **average Pass@16 improvements of** **1.7**, **1.7**, and **3.0 percentage points**, respectively, across the two models. These results provide further evidence that GMAE improves the likelihood of generating correct solutions.
We respectfully disagree with this concern. In the first paragraph of the Introduction, we use self-evolution as the broader research context and present self-rewarding RL as one approach to improving a policy through self-generated feedback on unlabeled prompts. We do not equate self-rewarding RL with all forms of self-evolution or claim it is necessary for them. We believe our framing does not constitute overclaiming.
We respectfully disagree that the reproducibility details are insufficient.
Hyperparameters. Section 5.1 provides the key training configurations, including the GRPO backbone, batch size (64), training epochs (1), learning rate (1×10−6), warmup ratio (5%), rollout size (16 main + 16 auxiliary responses), and sampling settings. We also specify the GPU hardware (NVIDIA A100 80GB). The Evaluation subsection further reports the inference sampling parameters (temperature 0.6, top-p 0.95, and top-k 20) and checkpoint selection protocol. If any details are still missing, we would appreciate it if you could point them out specifically.
Code Availability. We have included the anonymized code in the supplementary materials and will make the code publicly available upon acceptance of the paper.
Hallucinated References. We strongly disagree with the allegation of hallucinated references. All bibliographic entries were manually retrieved from Google Scholar by us. We believe such a serious allegation should be supported by concrete evidence and would appreciate it if you could identify the specific problematic references.
Font Size. We have enlarged the fonts in Table 1 and Figure 2 in the revised manuscript to improve readability, while strictly adhering to the original 9-page submission limit.