- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to combine heterogeneous reward signal (individual feedback sources have different limitations) in LLM alignment task. THis paper combines an endogenous score, a discriminative reward head and a generative verifier.The three scores are standardized and combined using inverse-dispersion weights to provide the terminal reward for PPO. This work also provide theoretical guarantees by proving the regret bound and evaluates the framework on different tasks.
Critical strengths
The three reward sources have their clear roles and fit into a straightforward training pipeline. It makes sense to me to combine preference discrimination, external verification, and SFT-based confidence, as they provide different perspectives on response quality. The shared-backbone design is also practical.
Critical weaknesses
I am not fully convinced by some parts of the method and experiments for the reasons as follows:
First, I think the weighting rule in Eq. (8) needs more justification. The authors seem to assign smaller weights to reward sources with larger output variance, but it is not clear to me why larger variance necessariily indicates that reward signal is less useful. I think it could also mean that the reward model is better at distinguishing different responses. Moreover, since the reward scores are already standardized before fusion, I am not sure why their original variance should determine the final weights.One example I can think is (please correct me if I misunderstood): suppose two reward models individually assign scores of (1, 2, 3) and (10, 20, 30) to the same three responses. They provide exactly the same information after standardization, but Eq. (8) assigns the first model nearly 100 times the weight of the second simply because its original scores have a smaller variance. I am not sure why such a difference in scale should affect the fusion weights. I think the reward design is the main contribution for this paper, so I would like to see more justification or experimental evidence could supports this design. (ps: It is good to see some theoretical guarantees connecting reward estimation errors to regret. However, I think the more important question is how to design the weighting rule in Eq. (8). The current analysis mainly explains how estimation errors propagate through the weighted reward, but does not justify why inverse-dispersion weighting is a good choice.)
Second, I have some concerns about the RM-Bench evaluation in Table 1. The paper reports results across different domains and difficulty levels, but it is not clear how the Avg column is computed from these results (PS: It seems to me that some rows' results are not consistent). This should be clarified.
Questions for authors
See the ’Weakness‘ section.
-
Could the authors provide more justification for the inverse-dispersion weighting in Eq. (8)? (It could be better if the authors could further evaluate its sensitivity to reward-score rescaling.)
-
How is the Avg column in Table 1 computed, and why do some reported averages appear inconsistent with the domain scores?
- I also feel interested in the robustness of the reward design. Could you compare the robustness of adaptive weighting withequal and frozen weights under the same reward perturbations?
4 comments
Response to Question 3
We compare adaptive, equal, and frozen weighting under the same Gaussian reward noise . Adaptive weights are updated online, while equal and frozen weights remain fixed. The table below reports the mean standard deviation under clean and perturbed rewards. Adaptive weighting achieves higher perturbed means than frozen weighting on all four benchmarks. Compared with equal weighting, its gains are 6.71%, 11.80%, and 1.80% on HumanEval, MBPP, and MATH-500, respectively, which tie at 18.67% on AIME 2024. Thus, adaptive weighting achieves the highest or joint-highest mean under the tested perturbation. We have added this comparison and clarified its scope in Section 5.6 of the revised manuscript.
| Weighting | Condition | HumanEval | MBPP | MATH-500 | AIME 2024 |
|---|---|---|---|---|---|
| Adaptive | Clean | ||||
| Perturbed | |||||
| Equal | Clean | ||||
| Perturbed | |||||
| Frozen | Clean | ||||
| Perturbed |
Reference
[1]. Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi-Hua Zhou. Generalist reward models: Found inside large language models. arXiv preprint arXiv:2506.23235, 2025. https://arxiv.org/abs/2506.23235
Thank you again for your efforts to review our paper. As the discussion period ends in one day, we would be grateful to know whether you have further concerns.
Thank you for the detailed response. My second and third questions have been addressed.
However, my main concern is still the weighting rule in Eq. (8). With the original scores, adaptive weighting works well. But scaling the endogenous reward by 0.1 leaves the responses in the same order and still lowers the MBPP score from 75.28 to 65.68. It is still unclear to me why a reward should get more weight just because its original scores are closer together. Therefore I will keep my score.
We thank the reviewer for carefully reading our paper and recognizing the complementary roles of our reward sources and the practical shared-backbone design. We address the reviewer's concerns below.
Response to Weakness 1 and Question 1
We address the motivation, scale sensitivity, and supporting evidence in three parts:
Inverse-dispersion weighting: Our motivation is that reducing the contribution of a reward stream may stabilize learning when its larger fluctuations reflect noise. However, output variance alone cannot distinguish noise from useful discrimination, so reducing a stream's weight is not always beneficial. Developing weighting rules that distinguish these sources of variation remains an important direction for future work. We have clarified this motivation in Section 3.2 and discussed the limitation and future direction in Section 6 of the revised manuscript.
Scale sensitivity: Eq. (8) depends on the raw score scale. For the same candidate responses, multiplying stream k by c>0 leaves its z-scores approximately unchanged if its statistics are scaled consistently and the numerical stabilizer is negligible. However, its weight relative to an unscaled stream j changes as follows:
αj′αk′=c2σk2+εσj2+ε≈c−2αjαk.
Thus, tenfold scaling reduces the relative weight to approximately 1/100 of its original value. In the reviewer's two-stream example, identical standardized scores yield the same fused reward despite the different weights, while heterogeneous streams need not have this property.
To evaluate this effect, we multiply each raw reward stream individually by 0.1 and 10 before normalization, leaving the other two streams unscaled. The table below compares these six conditions with the unscaled baseline under matched training and evaluation settings. Relative to this baseline, the largest absolute changes in mean performance are 8.42%, 9.60%, 2.16%, and 7.33% on HumanEval, MBPP, MATH-500, and AIME 2024, respectively. These results confirm substantial scale sensitivity. Besides, all six rescalings reduce the mean on MATH-500. We have clarified this scale dependence in Section 3.2 and added Table 5 to Section 5.5 of the revised manuscript.
Response to Weakness 2 and Question 2
We apologize for the unclear aggregation protocol in Table 1. Following the evaluation setup of EndoRM [1], we originally computed the LLM-as-a-Judge and RALA scores by weighting domains by sample count. We have corrected the LLM-as-a-Judge results to follow the official RM-Bench protocol and additionally report RALA under this protocol. Specifically, each difficulty score averages the four domains equally, and Avg averages the three difficulty scores. Under this protocol, RALA's Easy, Normal, Hard, and Avg scores are 75.6%, 70.4%, 59.7%, and 68.6%, respectively.
We retain RALA's sample-weighted results for comparison with the published EndoRM results and use its results under the official protocol for comparison with the other baselines evaluated under that protocol. RALA achieves the highest Hard score within each respective comparison group. We have updated Table 1 to distinguish these two groups and clarified their aggregation rules in Section 5.2 of the revised manuscript.