acceptodds
Under review as a conference paper at ICLR 2027

RALA: Reliability-Aware Latent Alignment under Heterogeneous Feedback

Abstract

Aligning Large Language Models (LLMs) with human intent is commonly achieved through Reinforcement Learning (RL) from Artificial Intelligence (AI) or human feedback. This approach typically relies on Reward Models (RMs) to transform preference feedback into training signals, making RMs a critical bottleneck in LLM alignment. Existing methods primarily rely on a single feedback source and do not account for the heterogeneous reliability and bias of real-world supervision. This paper interprets LLM alignment as a latent reward inference problem under heterogeneous and unreliable feedback. The true alignment signal is modeled as a latent reward, with feedback from different sources serving as noisy and potentially biased observations thereof. We propose *reliability-aware latent alignment (RALA)*, a unified framework that combines discriminative, generative, and endogenous sequence-level reward estimators. Its discriminative branch reuses the SFT backbone attaching a lightweight and gradient-isolated reward head. RALA uses inverse-dispersion weighting to reduce the influence of reward components with large output fluctuations during policy optimization. Our theoretical analysis establishes finite-sample regret bounds, linking feedback reliability and component-wise reward estimation errors to downstream policy performance. Extensive experiments on benchmarks spanning reward modeling, code generation, and mathematical reasoning demonstrate that RALA improves alignment performance under mixed-quality feedback. Our code is available at https://anonymous.4open.science/r/RALA-F983.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Alignment, Reward Modeling, Latent Variable Learning, Feedback Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xxmZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to combine heterogeneous reward signal (individual feedback sources have different limitations) in LLM alignment task. THis paper combines an endogenous score, a discriminative reward head and a generative verifier.The three scores are standardized and combined using inverse-dispersion weights to provide the terminal reward for PPO. This work also provide theoretical guarantees by proving the regret bound and evaluates the framework on different tasks.

Critical strengths

The three reward sources have their clear roles and fit into a straightforward training pipeline. It makes sense to me to combine preference discrimination, external verification, and SFT-based confidence, as they provide different perspectives on response quality. The shared-backbone design is also practical.

Critical weaknesses

I am not fully convinced by some parts of the method and experiments for the reasons as follows:

First, I think the weighting rule in Eq. (8) needs more justification. The authors seem to assign smaller weights to reward sources with larger output variance, but it is not clear to me why larger variance necessariily indicates that reward signal is less useful. I think it could also mean that the reward model is better at distinguishing different responses. Moreover, since the reward scores are already standardized before fusion, I am not sure why their original variance should determine the final weights.One example I can think is (please correct me if I misunderstood): suppose two reward models individually assign scores of (1, 2, 3) and (10, 20, 30) to the same three responses. They provide exactly the same information after standardization, but Eq. (8) assigns the first model nearly 100 times the weight of the second simply because its original scores have a smaller variance. I am not sure why such a difference in scale should affect the fusion weights. I think the reward design is the main contribution for this paper, so I would like to see more justification or experimental evidence could supports this design. (ps: It is good to see some theoretical guarantees connecting reward estimation errors to regret. However, I think the more important question is how to design the weighting rule in Eq. (8). The current analysis mainly explains how estimation errors propagate through the weighted reward, but does not justify why inverse-dispersion weighting is a good choice.)

Second, I have some concerns about the RM-Bench evaluation in Table 1. The paper reports results across different domains and difficulty levels, but it is not clear how the Avg column is computed from these results (PS: It seems to me that some rows' results are not consistent). This should be clarified.

Questions for authors

See the ’Weakness‘ section.

  1. Could the authors provide more justification for the inverse-dispersion weighting in Eq. (8)? (It could be better if the authors could further evaluate its sensitivity to reward-score rescaling.)

  2. How is the Avg column in Table 1 computed, and why do some reported averages appear inconsistent with the domain scores?


  1. I also feel interested in the robustness of the reward design. Could you compare the robustness of adaptive weighting withequal and frozen weights under the same reward perturbations?
4 comments

We thank the reviewer for carefully reading our paper and recognizing the complementary roles of our reward sources and the practical shared-backbone design. We address the reviewer's concerns below.

Response to Weakness 1 and Question 1

We address the motivation, scale sensitivity, and supporting evidence in three parts:

  • Inverse-dispersion weighting: Our motivation is that reducing the contribution of a reward stream may stabilize learning when its larger fluctuations reflect noise. However, output variance alone cannot distinguish noise from useful discrimination, so reducing a stream's weight is not always beneficial. Developing weighting rules that distinguish these sources of variation remains an important direction for future work. We have clarified this motivation in Section 3.2 and discussed the limitation and future direction in Section 6 of the revised manuscript.

  • Scale sensitivity: Eq. (8) depends on the raw score scale. For the same candidate responses, multiplying stream kk by c>0c>0 leaves its z-scores approximately unchanged if its statistics are scaled consistently and the numerical stabilizer is negligible. However, its weight relative to an unscaled stream jj changes as follows:

αk′αj′=σj2+εc2σk2+ε≈c−2αkαj.\frac{\alpha'_k}{\alpha'_j} = \frac{\sigma_j^2+\varepsilon}{c^2\sigma_k^2+\varepsilon} \approx c^{-2}\frac{\alpha_k}{\alpha_j}.

Thus, tenfold scaling reduces the relative weight to approximately 1/1001/100 of its original value. In the reviewer's two-stream example, identical standardized scores yield the same fused reward despite the different weights, while heterogeneous streams need not have this property.

To evaluate this effect, we multiply each raw reward stream individually by 0.10.1 and 1010 before normalization, leaving the other two streams unscaled. The table below compares these six conditions with the unscaled baseline under matched training and evaluation settings. Relative to this baseline, the largest absolute changes in mean performance are 8.42%, 9.60%, 2.16%, and 7.33% on HumanEval, MBPP, MATH-500, and AIME 2024, respectively. These results confirm substantial scale sensitivity. Besides, all six rescalings reduce the mean on MATH-500. We have clarified this scale dependence in Section 3.2 and added Table 5 to Section 5.5 of the revised manuscript.

Reward transformationHumanEvalMBPPMATH-500AIME 2024
Original rewards82.93±0.9682.93\pm0.9675.28±1.5575.28\pm1.5572.80±0.1472.80\pm0.1422.00±1.8322.00\pm1.83
Discriminative reward ×0.1\times0.177.56±0.8077.56\pm0.8066.12±2.2366.12\pm2.2371.12±0.3371.12\pm0.3314.67±2.9814.67\pm2.98
Discriminative reward ×10\times1080.73±1.4080.73\pm1.4073.88±1.2373.88\pm1.2371.28±0.4171.28\pm0.4114.67±3.8014.67\pm3.80
Generative reward ×0.1\times0.175.24±1.4775.24\pm1.4767.48±1.8067.48\pm1.8071.00±0.3771.00\pm0.3714.67±3.8014.67\pm3.80
Generative reward ×10\times1083.29±1.8183.29\pm1.8173.88±2.2373.88\pm2.2370.64±0.3370.64\pm0.3323.33±3.3323.33\pm3.33
Endogenous reward ×0.1\times0.174.51±1.4674.51\pm1.4665.68±1.1565.68\pm1.1570.96±0.7570.96\pm0.7520.67±1.4920.67\pm1.49
Endogenous reward ×10\times1080.73±1.2680.73\pm1.2672.92±1.7572.92\pm1.7571.24±0.2671.24\pm0.2620.00±2.3620.00\pm2.36
  • Theoretical scope and empirical support: Our theory relates component estimation errors and feedback bias to policy regret under convex reward fusion, which does not derive inverse-dispersion weighting. The controlled comparisons in Table 4 of Section 5.5 provide empirical support under the original reward scales. Adaptive weighting achieves the highest mean on all four benchmarks, with MATH-500 gains of 1.92% and 1.56% over equal and frozen weights, respectively. These results support the tested configuration, while the rescaling study shows its dependence on numerical reward scales. We have clarified the distinction between theoretical guarantees and empirical evidence in Sections 3.2 and 4 of the revised manuscript.

Response to Weakness 2 and Question 2

We apologize for the unclear aggregation protocol in Table 1. Following the evaluation setup of EndoRM [1], we originally computed the LLM-as-a-Judge and RALA scores by weighting domains by sample count. We have corrected the LLM-as-a-Judge results to follow the official RM-Bench protocol and additionally report RALA under this protocol. Specifically, each difficulty score averages the four domains equally, and Avg averages the three difficulty scores. Under this protocol, RALA's Easy, Normal, Hard, and Avg scores are 75.6%, 70.4%, 59.7%, and 68.6%, respectively.

We retain RALA's sample-weighted results for comparison with the published EndoRM results and use its results under the official protocol for comparison with the other baselines evaluated under that protocol. RALA achieves the highest Hard score within each respective comparison group. We have updated Table 1 to distinguish these two groups and clarified their aggregation rules in Section 5.2 of the revised manuscript.

Response to Question 3

We compare adaptive, equal, and frozen weighting under the same Gaussian reward noise N(0,0.01)\mathcal{N}(0,0.01). Adaptive weights are updated online, while equal and frozen weights remain fixed. The table below reports the mean ±\pm standard deviation under clean and perturbed rewards. Adaptive weighting achieves higher perturbed means than frozen weighting on all four benchmarks. Compared with equal weighting, its gains are 6.71%, 11.80%, and 1.80% on HumanEval, MBPP, and MATH-500, respectively, which tie at 18.67% on AIME 2024. Thus, adaptive weighting achieves the highest or joint-highest mean under the tested perturbation. We have added this comparison and clarified its scope in Section 5.6 of the revised manuscript.

WeightingConditionHumanEvalMBPPMATH-500AIME 2024
AdaptiveClean82.93±0.9682.93\pm0.9675.28±1.5575.28\pm1.5572.80±0.1472.80\pm0.1422.00±1.8322.00\pm1.83
Perturbed77.32±1.1777.32\pm1.1771.76±1.5171.76\pm1.5171.28±0.2371.28\pm0.2318.67±1.8318.67\pm1.83
EqualClean76.34±1.1776.34\pm1.1764.04±0.3664.04\pm0.3670.88±0.3670.88\pm0.3620.67±2.7920.67\pm2.79
Perturbed70.61±0.8070.61\pm0.8059.96±1.6359.96\pm1.6369.48±0.3069.48\pm0.3018.67±2.9818.67\pm2.98
FrozenClean80.49±1.2280.49\pm1.2273.88±1.0073.88\pm1.0071.24±0.0971.24\pm0.0918.67±1.8318.67\pm1.83
Perturbed75.85±1.7675.85\pm1.7669.80±1.8069.80\pm1.8069.08±0.2769.08\pm0.2716.00±1.4916.00\pm1.49

Reference

[1]. Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi-Hua Zhou. Generalist reward models: Found inside large language models. arXiv preprint arXiv:2506.23235, 2025. https://arxiv.org/abs/2506.23235

Thank you again for your efforts to review our paper. As the discussion period ends in one day, we would be grateful to know whether you have further concerns.

Thank you for the detailed response. My second and third questions have been addressed.

However, my main concern is still the weighting rule in Eq. (8). With the original scores, adaptive weighting works well. But scaling the endogenous reward by 0.1 leaves the responses in the same order and still lowers the MBPP score from 75.28 to 65.68. It is still unclear to me why a reward should get more weight just because its original scores are closer together. Therefore I will keep my score.

Reviewer_6zBmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RALA fuses three types of response-level rewards in a novel method. First, there is a gradient-isolated discriminative head based on SFT architecture; second, there is a length-normalized SFT likelihood score; third, there is the probability of the positive verdict by the verifier network. All three types of rewards are standardized before fusion. The variance of pre-standardization is used for dynamic adjustment of the fusion coefficients via PPO algorithm. The authors conduct theoretical analysis and transfer component-learning guarantee, and introduce a feedback contamination bias term to policy regret bounds with faster rates with marginal and gap assumptions. The experiments include RM-Bench dataset and tasks of code generation, and mathematics reasoning. In summary, the unique feature of this paper lies in the specific combination scheme, experimental results, and conditional error propagation analysis. The scoring interfaces themselves have precedents in EndoRM and Generative Verifiers respectively.

Critical strengths

The most convincing strength of this work is the controlled ablation testings conducted in Table 3 and Table 4. With regard to the MATH-500 task, RALA obtains 72.80 ± 0.14, surpassing the best mean performance of any single component (71.16 ± 0.26). Besides, it improves over equal weight fusion (70.88 ± 0.36) and fixed converged weight fusion (71.24 ± 0.09) significantly. The reporting of five-seed sample standard deviation makes within-setting comparison more convincing.

Another convincing point is the detailed theoretical analyses presented in Appendix B. It explicitly considers possible correlations among components’ errors, distinguishes the probabilities from reward-coordinates in analysis, and further extends the theory in terms of target mismatch and optimization gap scenarios. I haven’t discovered any fatal flaw in the main argumentation process concerning error-transferring.

Critical weaknesses

The primary weakness of this paper relates to the core claims made in Section 3 about adaptive weighting. The results of freezing weights of Code (see Table C.2.4 of Appendix) yield (0.3426, 0.3309, 0.3265) which is almost equivalent to equal weight distribution (0.3333, 0.3333, 0.3333). Yet, there exists a 9.84-points MBPP gap (73.88% versus 64.04%). Despite the use of z-score normalization in Eq. (8) and relatively stable trajectories in Fig. 3, this huge gap still poses doubts about the necessity of adaptive weighting scheme online. Additionally, the authors don’t provide sufficient evidence suggesting that raw dispersion measures of rewards correspond with reliability in terms of feedback reliability. Indeed, the theoretical result is for general convex combination coefficient and doesn’t prove any reliability-based justification for inverse-dispersion weighing.

Furthermore, in relation to the headline gain of mathematics task performance, the baseline difference issue needs addressing. Specifically, on MATH-500, original SFT model yields 40.0% whereas all single-reward models obtain roughly 71%. Meanwhile, full RALA obtains 72.8% which is merely 1.64 points higher than the strongest single-stream baseline. Additionally, comparing Tables 2 & 3, there are huge gaps in performance between generative-only (55.6% versus 71.0%) and endogenous-only baselines (67.4% versus 70.68%). Those huge gaps raise suspicion of whether the improvements can be reasonably attributed to generic RL optimizations instead.

Questions for authors

Eq. (7) uses the next-token probability of "Yes" as the generative reward, without explicit reasoning or a reference solution in the verification prompt (Appendix D.1). How reliably can this approach detect subtle errors in multi-step mathematical solutions? I am strongly suspicious that this approach can be adopted in challenging cases where multi step reasoning is necessary.

How to deal with correct self-generated responses, as well as ties in the pair-wise label generation process? Especially in the case of Appendix E.2 (p. 32), both stamp-solutions and solution produce the right answer 9. How are relative preference ratings transformed into Yes/No labeling?

How does the alternative approach mentioned in Table 3 replace representation-level processing? Do the datasets, model capacity, and optimizations remain identical?

4 comments

We thank the reviewer for carefully reading our paper and recognizing the strengths of our controlled ablations and theoretical analysis. We address the reviewer's concerns below.

Response to Weakness 1

We address these concerns in two parts:

  • Online adaptation: Small weight changes may reverse candidate rankings when reward streams disagree and fused scores are close, leading to different PPO advantages and subsequent rollouts. In Table 4, the frozen baseline uses the final coefficients from a reference adaptive run and holds them fixed throughout PPO training. Adaptive weighting achieves 75.28±1.55%75.28 \pm 1.55\% on MBPP, compared with 73.88±1.00%73.88 \pm 1.00\% for frozen weighting, a gain of 1.40%. This supports the value of online updates relative to the tested fixed mixture. We have clarified these comparisons in Section 5.5 of the revised manuscript.

    We also evaluate adaptive, equal, and frozen weighting with Gaussian reward noise N(0,0.01)\mathcal{N}(0,0.01). As shown below, adaptive weighting achieves higher mean scores than frozen weighting on all four benchmarks under this perturbation. Its advantages over frozen weighting on MBPP and MATH-500 increase from 1.40% and 1.56% under clean rewards to 1.96% and 2.20% under perturbed rewards. These results support the practical value of online updates under the tested perturbation. We have added this comparison to Section 5.6 of the revised manuscript.

WeightingConditionHumanEvalMBPPMATH-500AIME 2024
AdaptiveClean82.93±0.9682.93 \pm 0.9675.28±1.5575.28 \pm 1.5572.80±0.1472.80 \pm 0.1422.00±1.8322.00 \pm 1.83
Perturbed77.32±1.1777.32 \pm 1.1771.76±1.5171.76 \pm 1.5171.28±0.2371.28 \pm 0.2318.67±1.8318.67 \pm 1.83
EqualClean76.34±1.1776.34 \pm 1.1764.04±0.3664.04 \pm 0.3670.88±0.3670.88 \pm 0.3620.67±2.7920.67 \pm 2.79
Perturbed70.61±0.8070.61 \pm 0.8059.96±1.6359.96 \pm 1.6369.48±0.3069.48 \pm 0.3018.67±2.9818.67 \pm 2.98
FrozenClean80.49±1.2280.49 \pm 1.2273.88±1.0073.88 \pm 1.0071.24±0.0971.24 \pm 0.0918.67±1.8318.67 \pm 1.83
Perturbed75.85±1.7675.85 \pm 1.7669.80±1.8069.80 \pm 1.8069.08±0.2769.08 \pm 0.2716.00±1.4916.00 \pm 1.49
  • Dispersion and reliability: Yes, output variance can reflect noise, scale differences, or useful discrimination. However, our motivation is that, when larger fluctuations contain more noise, reducing their contribution may provide a more stable learning signal. The theory relates component errors and feedback bias to policy regret under convex reward fusion, which does not derive inverse-dispersion weighting or establish its optimality. We have clarified this motivation and the scope of the theory in Sections 3.2 and 4, and discussed the limitation and future work on distinguishing noise from useful discrimination in Section 6 of the revised manuscript.

Response to Weakness 2

We evaluate reward fusion under matched policy initialization, training data, and RL settings within each task in Table 3:

  • Gains from reward fusion: The difference between 40.0% and 72.8% involves both a change in policy initialization and RL training, so it does not isolate the gain from RL or reward fusion. Under the matched conditions of Table 3, RALA achieves 72.80±0.14%72.80 \pm 0.14\% on MATH-500, compared with 71.16±0.26%71.16 \pm 0.26\% for the strongest single-reward baseline over five seeds. This 1.64% gain supports the contribution of reward fusion within the same RL training procedure. We have clarified this distinction in Section 5.3 of the revised manuscript.

  • Differences between Tables 2 and 3: We apologize for not clearly specifying the result sources and policy initializations. In Table 2, the MATH-500 scores of 55.6% for Generative Verifier and 67.4% for EndoRM are taken from EndoRM [1]. These two math baselines and LLM-as-a-Judge use Qwen2.5-7B-Math. Table 3 reports our controlled ablations, where all math variants use Qwen2.5-7B-Math-Instruct, as does RALA in Table 2. Therefore, these cross-table score differences involve different initialization settings and do not isolate the effect of reward fusion. We have identified the externally reported scores and policy initializations in Table 2 and clarified the experimental setup in Section 5.1.

    For a fair comparison, we have additionally evaluated all three baselines using the same Qwen2.5-7B-Math-Instruct initialization as RALA. The results are reported in Table 10 of Appendix C.3.3 and summarized in the table below. With matched initialization, RALA achieves the highest scores on both benchmarks, exceeding the strongest baseline by 1.6% on MATH-500 and 2.0% on AIME 2024.

MethodMATH-500AIME 2024
RLAIF with LLM-as-a-Judge69.416.7
RLAIF with Generative Verifier71.220.0
RLHF with EndoRM70.820.0
RALA (Proposed)72.822.0

Response to Question 1

RALA combines three complementary reward signals. The discriminative RM scores a response representation that incorporates intermediate reasoning steps. The generative RM also receives the question and candidate response, including any intermediate reasoning steps, and is asked whether the response is "correct and complete", as shown in Appendix D.1. The Yes/No format specifies its output and does not restrict its input to the final answer. Appendix F.4 illustrates multi-step solutions generated for AIME 2024. We have clarified these reward roles in Section 3.2 and Appendix D.1 of the revised manuscript.

Response to Question 2

The discriminative and generative RMs use different labeling procedures:

  • Correct self-generated responses: Both solutions in the stamp example in Appendix E.2 obtain the correct answer of 9. These are discriminative RM training pairs, rather than generative RM verification examples. As described in Section 3.1, reference solutions are assigned to the positive position and policy-sampled responses to the negative position, without external correctness filtering. Thus, "negative" denotes the response's position in the ranking pair, rather than verified incorrectness. During policy optimization, this discriminative score is combined with the endogenous and generative rewards through Eq. (8).

  • Preference ties: For AI-generated preferences in Appendix D.2, we evaluate each candidate pair (A,B)(A,B) in both presentation orders (AB and BA) and average the directional preference probabilities to reduce order sensitivity. Any exact probability tie after averaging is resolved deterministically in favor of candidate AA. For externally supplied preference datasets, we follow the provided chosen/rejected ordering without a separate tie class.

  • Conversion to Yes/No labels: Under the prompt in Appendix D.1, each ordered preference pair yields two verification examples: the chosen response is labeled "Yes" and the rejected response "No". The generative RM is trained with binary classification and pairwise ranking losses. Its output, pϕ("Yes"∣x,y,P)p_\phi(\text{"Yes"} \mid x,y,\mathcal{P}), is standardized and combined with the other rewards in Eq. (8). These labels encode relative preference rather than independent correctness annotations, so a correct but less preferred response can receive "No". Definition 4.1 models imperfect generative supervision through a clean-noisy feedback mixture, and Theorem 4.2 analyzes its contribution to policy regret. We have clarified these labeling conventions and their scope in Sections 3.1–3.2 and Appendix E.2 of the revised manuscript.

Response to Question 3

In Table 3, the "w/o Representation-level processing" variant trains the discriminative RM on features after the token head. RALA uses the final response-token hidden state before the token head. Both variants use the same datasets and training settings. We have clarified this configuration in Section 5.4 and Appendix C.2.1 of the revised manuscript.


Reference

[1]. Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi-Hua Zhou. Generalist reward models: Found inside large language models. arXiv preprint arXiv:2506.23235, 2025. https://arxiv.org/abs/2506.23235

Thank you again for your efforts to review our paper. As the discussion period ends in one day, we would be grateful to know whether you have further concerns.

Thank you for the response. I think weakness 2 is basically resolved. Critical concerns remain: The authors didn't fully explain why small weight changes effect MBPP performance so significantly, more than all baseline comparisons (73.88 vs 64.04, compared with ~2pp scale usually used). If 'small weight changes may lead to high fluctuation in PPO/rollouts/downstream score, why the performance is so stable between seeds(only variation of 0.36 fand 1.00 for these two results)? It makes me even more confusing that in Table 5, huge weight change caused tiny change on MBPP (73.88 vs 73.88). Plus, the authors may misunderstand my previous Question 1. I mean, the judging task is hard for RM, like challenging math/coding tasks. But RM is asked to give directly give Yes/No judgement, without any thinking process during its judgement. It may be highly unreliable. Since the results presented in this work seem highly unstable and unlikely fully addressed during the rebuttal period, I would keep my initial score and encourage the authors to strictly audit the results to make it better.

Reviewer_9UqSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to answer the question that whether rewards from multiple sources can be utilized together to produce better rewards, although each of them is imperfect alone. The paper models the problem as latent rewards learning and uses dynamic weighting to project the multiple rewards to it. The paper conducts experiments and evaluates on popular reward benchmarks, and some code and math benchmarks.

Critical strengths
  1. The idea of projecting 3 reward sources with quite differences to one latent reward is interesting.
  2. The paper uses a lot of pages to model the theoretical framework and seems to have no obvious issues.
Critical weaknesses
  1. Important experimental details are quite unclear or missing. The settings like the base models, training dataset or baselines seem to vary across context, which is ambiguous and prevents me from assessing the true improvement of the proposed framework. For example, math SFT data, reward-training data, PPO prompt is not specified. Table 2 has a different base model, which conflicts Implementation Details in the setup. Qwen2.5-7B-Math seems to be called as Qwen2.5-7B-Math-Instruct somewhere as the same model.
  2. The presentation is awful and unpolished. For example, as one of the most important table, Table 2 contains GPT-5.2 and ChatGPT at the same time. What is ChatGPT here? Also the definition/details of the used Generative Verifier within the proposed framework is missing, while appearing in multiple tables.
Questions for authors

A lot of work seems to be spent on the theoretical part, how do you define this work?

4 comments

We thank the reviewer for their careful reading and constructive feedback. We address the reviewer's concerns below.

Response to Weakness 1

We apologize for the missing details and clarify the experimental setup in three parts:

  • Training data: For math reasoning, we initialize RALA's policy from Qwen2.5-7B-Math-Instruct without additional SFT. We train the discriminative and generative RMs on MetaMathQA (Yu et al., 2024) and Math-Step-DPO-10K (Lai et al., 2024), respectively, and use all 7,500 training questions from MATH (Hendrycks et al., 2021) as PPO prompts. The code SFT data and reward-training data construction are already described in Section 5.1 and Appendix C.1 of the submitted manuscript. We have added the missing math training details to these sections in the revised manuscript.

  • Comparisons in Table 2: Table 2 presents off-the-shelf reference systems in the upper block and results for alignment methods in the lower block. The MATH-500 scores for Generative Verifier and EndoRM are taken from EndoRM (Li et al., 2025). We have identified these externally reported scores and clarified the distinction between the two blocks in the caption of Table 2.

  • Model initialization: Qwen2.5-7B-Math and Qwen2.5-7B-Math-Instruct are distinct checkpoints. RALA uses Qwen2.5-7B-Math-Instruct for both RM-Bench and math reasoning, and DeepSeek-Coder-6.7B for code generation. The LLM-as-a-Judge, Generative Verifier, and EndoRM math baselines in Table 2 use Qwen2.5-7B-Math, following the model choice in EndoRM (Li et al., 2025). The controlled comparisons in Table 3 share initialization and training data within each task, while all math variants use Qwen2.5-7B-Math-Instruct. We have clarified these checkpoint choices in Section 5.1 and Table 2.

Response to Weakness 2

We clarify the model names and the verifier definition as follows:

  • ChatGPT and GPT-5.2: "ChatGPT" in Table 2 refers to ChatGPT-4o, which is distinct from GPT-5.2 (OpenAI model documentation). We have replaced "ChatGPT" with "ChatGPT-4o" in Table 2 to make the model identity explicit.

  • Generative Verifier: We use the Generative Verifier proposed by Zhang et al. (2025), which scores a candidate response using the next-token probability of "Yes" given the question and response, as defined in Eq. (7). Its training data construction, hyperparameters, and verification prompt are described in Appendices C.1, C.2.2, and D.1, respectively. The Generative Verifier baseline prompt is provided in Appendix D.3.

Response to Question 1

RALA is a method for integrating complementary and individually imperfect reward signals for LLM alignment. Its theoretical contribution is a finite-sample analysis of how reward estimation errors and feedback bias affect downstream policy regret.

Under explicit common-target and component-learning assumptions, our theory bounds the fused-reward error in terms of component errors and relates this bound to policy regret. The analysis also separates reward errors from policy optimization errors, providing guidance for improving reward models and policy training.

The practical implication is that feedback quantity and quality affect different terms in the bound. More samples reduce the estimation terms, while the bias term of a fixed feedback source is not reduced by collecting more samples from that source. Assuming a smaller bias term for human feedback, Corollary B.8 in Appendix B.3 compares regret upper bounds for expanding AI-generated feedback versus obtaining more reliable human supervision. This provides a theoretical reference for allocating annotation resources.

We have revised the manuscript to clarify RALA's methodological contribution and the theoretical and practical implications of the analysis.

References

[1]. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

[2]. Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X., and Jia, J. (2024). Step-DPO: Step-wise preference optimization for long-chain reasoning of LLMs. arXiv preprint arXiv:2406.18629. https://arxiv.org/abs/2406.18629

[3]. Li, Y.-C., Xu, T., Yu, Y., Zhang, X., Chen, X.-H., Ling, Z., Chao, N., Yuan, L., and Zhou, Z.-H. (2025). Generalist reward models: Found inside large language models. arXiv preprint arXiv:2506.23235. https://arxiv.org/abs/2506.23235

[4]. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. (2024). MetaMath: Bootstrap your own mathematical questions for large language models. The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2309.12284

[5]. Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. (2025). Generative verifiers: Reward modeling as next-token prediction. The Thirteenth International Conference on Learning Representations.

Thank you again for your efforts to review our paper. As the discussion period ends in one day, we would be grateful to know whether you have further concerns.

Reviewer_9UqSOpenReview ↗

Thanks for your rebuttal.

Thank you for clarifying the math training setup and adding Table 10. I have three remaining points.

  1. Table 2. RALA starts from Qwen2.5-7B-Math-Instruct, while three math baselines start from Qwen2.5-7B-Math. This makes it difficult to attribute the score differences to the alignment method. With matched checkpoints in Table 10, the MATH-500 gap between RALA and the Generative Verifier narrows from 17.2 to 1.6 percentage points. Could you make the matched results the primary comparison and mark which Table 2 scores come from prior work?
  2. Table 1. My understanding is that the Generative Verifier score comes from Li et al.’s training-free Qwen2.5-7B-Instruct verifier. Could you identify that checkpoint in the table and clarify whether you also evaluated RALA’s own trained verifier alone on RM-Bench? That comparison would help isolate what the other two reward signals add.
  3. ChatGPT-4o. Which model or evaluation produced this row? If you ran the evaluation, please give the API model ID or specify the ChatGPT interface and evaluation date. If the scores come from prior work, please cite the source.

Overall, I think the manuscript would benefit from substantial polishing in terms of writing quality, presentation, table clarity, and the design and description of the experiments. I therefore maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.