acceptodds
Under review as a conference paper at ICLR 2027

Learning From Everything: Dense Rewards From Every Kind of Feedback

Abstract

Learning jointly from different kinds of feedback remains an open challenge in language model post-training. Beyond verifiable and rubric-based rewards, current pipelines rely primarily on demonstrations and preferences, leaving other useful signals underused. We propose a unified and scalable method for learning dense rewards jointly from demonstrations, preferences, edits, ratings, stops, and more. We model each feedback type through its own likelihood and use Bayesian inference to learn a shared distribution over token-level rewards. The resulting probabilistic reward model provides dense rewards with uncertainty estimates and accommodates any feedback that can be modeled through a likelihood over rewards or action values. Training a model from diverse feedback then reduces to learning one shared reward model and using its rewards to optimize the policy. On the mathematical reasoning and safety subsets of RewardBench, the best feedback combinations improve accuracy over preferences alone by up to 17.4 and 5.9 percentage points, respectively, and over standard sequence-level reward models by 19.8 and 17.8 points. The learned rewards also improve error localization within responses and support stronger, more sustained improvements during PPO on mathematical reasoning tasks. Together, these results demonstrate the value of a unified reward model for turning diverse feedback into better policies.

Primary area:
reinforcement learning
Keywords:
reinforcement learning from human feedback, rlhf, reward learning, diverse feedback, bayesian reward learning, reward models for llms, token-level rewards

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ejuxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MELT, a method for learning from different kinds of feedback including demonstrations, preferences, ratings, edits, and stops. It models each feedback through a feedback-specific likelihood and uses Bayesian inference to learn a shared distribution over token-level rewards. Its core idea is that each type of feedback provides different clues about what makes a response good or bad. The probabilistic reward model delivers rewards with uncertainty estimates, which are then used to optimize the policy. Experiments on mathematical reasoning and safety show that combining feedback types improves response ranking and error localization

Critical strengths

MELT combines demonstrations, preferences, edits, ratings, and stops through feedback-specific likelihoods. This allows different signals to inform one shared reward model while retaining their properties token-level rewards support error localization. The model can identify where a response goes wrong In mathematical reasoning, the paper holds the total number of feedback examples fixed. And provides evidence that complementary feedback helps beyond additional data alone The experiments examine different feedback combinations and report uncertainty/variations across runs

Critical weaknesses

It is unclear how much improvement comes from MELT itself. The scalar reward model uses only preferences, while MELT’s strongest results also use demonstrations and other feedback Adding more feedback types does not always help. MELT shows large gains on RewardBench math, but no clear improvement over the scalar reward model on RewardBench 2 math. In safety, combining all four feedback types performs worse than some combinations with fewer types The paper predicts a reward distribution for each token and shows its uncertainty, but does not establish whether these estimates reliably identify inaccurate predictions.

Questions for authors

does MELT outperform SFT followed by preference-based optimization? using the same demonstrations and preferences? What explains the limited gains on RewardBench 2 math and the weaker performance of the four-modality model in safety? Do higher predicted uncertainties correspond to less reliable reward predictions? How is the assumption that the observed successor action is near-optimal justified for trajectories containing errors?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ejux

Thank you for the constructive review. We appreciate your positive assessment of MELT’s formulation and experimental analysis. Your questions motivated additional comparisons with SFT→DPO, broader response-ranking evaluations, and an ablation of the TD target, which we discuss below.

W1: Uneven gains across benchmarks and feedback combinations.

RB2 Math*.* To assess whether the gains extend beyond RB1, we added best-of-16 evaluations on GSM8K, MATH500, MMLU, and MMLU-Pro, where all reward models rank the same candidates sampled from a shared policy. Each reward model selects its highest-scoring response; we report its accuracy (mean of 5 seeds):

Reward modelGSM8KMATH500MMLUMMLU-Pro
Scalar (P)41.410.140.122.0
MELT P38.98.938.020.9
MELT P+D43.212.243.222.4
MELT P+S42.79.440.121.9
MELT P+D+S45.110.943.523.7

MELT P+D and P+D+S outperform both preference-only MELT and the scalar model in mean accuracy on all four benchmarks. Preference-only MELT remains below the scalar model on all four benchmarks, supporting that these gains primarily come from combining feedback rather than the token-level formulation alone. These results are reported in Table 3, with all feedback configurations and standard errors in App. B.1.

Four-modality safety model. This model still exceeds preference-only MELT and the scalar model on RB1, and matches the scalar model on RB2. Its smaller gains relative to some combinations may reflect differences between feedback sources: edits come from CAI/HH-style revisions, while preferences and demonstrations come from Nemotron-Safety, with different prompts, response generators, and annotation criteria. Additional feedback may therefore offer limited benefit when its signals conflict with existing supervision. This is consistent with prior findings on feedback informativeness (Baur et al., 2026; Metz et al., 2025). Collecting multiple feedback types for the same responses would help isolate the effect of feedback format and we see this as a valuable direction for future work.

W2. Reliability of uncertainty estimates. Our experiments focus on the rewards learned through probabilistic multi-feedback inference and their usefulness for ranking and policy optimization. The posterior variance contributes to this process through TD-consistency weighting (Eq. 3) and reward sampling during PPO. Its usefulness as a separate reliability signal remains to be evaluated, as stated in the Sec. 6 of the original version of the paper.

Q1. Comparison with SFT followed by preference-based optimization. We ran an SFT->DPO baseline using the same demonstration and preference data and the same initial checkpoint as MELT. SFT->DPO uses demonstrations for additional math-specific SFT, while MELT uses them jointly with preferences to learn the reward model and starts PPO from the initial checkpoint. SFT and DPO checkpoints are selected by early stopping on the validation set; for MELT PPO, we report the final checkpoint (step 800).

MethodGSM8KMATH500MMLU-Pro Math
SFT40.0 ± 0.413.4 ± 0.720.8 ± 0.4
SFT → DPO40.9 ± 0.414.0 ± 0.820.6 ± 0.7
MELT P+D → PPO44.3 ± 0.414.2 ± 0.323.8 ± 0.3

Mean accuracy ± standard error over four runs. MELT achieves higher accuracy on GSM8K and MMLU-Pro Math, with similar performance on MATH500. These results support using demonstrations to inform reward learning alongside preferences, beyond their conventional use for SFT. These comparisons are in Sec. 5.4 (Fig. 4).

Q2. Applying TD consistency to all trajectories. Our ablations support the observed-action bootstrap as an effective practical approximation, with no consistent improvement from switching to a greedy maximum over the successor state. Although these continuations need not be optimal, the term can still provide a useful learning signal. We tested this approximation by replacing the observed successor value with $\max_{a'} Q_\phi(s_{t+1}, a')$, which drops the assumption that the recorded next action is near-optimal.

DomainFeedbackObservedGreedy max
MathP+D85.2 / 55.186.4 / 54.5
MathP+D+S94.3 / 51.890.8 / 55.7
SafetyP+D77.7 / 55.076.5 / 54.9
SafetyP+E74.5 / 56.571.8 / 56.0
SafetyP+D+E+R72.6 / 51.071.8 / 49.9

Entries report RewardBench 1 / RewardBench 2 accuracy, averaged over five seeds. Neither variant has a consistent advantage. Applying TD to demonstrations only gave worse results (RB2 Math: 50.5 vs. 55.1 for P+D), suggesting that TD on other feedback types' trajectories provides useful supervision in this setting. These results and a discussion of the assumption are in App. B.5 (Table 9).

Reviewer_iVcTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces MELT, a framework for learning dense rewards from various kinds of feedback, such as rating, edits and interventions. Instead of simply providing the response-level or process-level reward, MELT constructs the token-level reward model by modeling each kind of feedback through its own likelihood estimation. The token-level reward model outputs a reward distribution and is trained with an auxiliary value head. The learned dense rewards are then used to train the policy with PPO. Experiments demonstrate that on MELT can improve policy's mathematical reasoning and enhance safety alignment.

Critical strengths
  1. This paper investigates how to leverage feedback beyond simple preference annotation, such as edits, ratings, and stops, which is very practical.
  2. This paper provides relatively solid technical contributions to building a unified token-level reward framework and learning dense rewards from different kinds of feedback.
  3. The proposed token-level reward model is effective when integrated with PPO and improves model performance.
Critical weaknesses
  1. The experiments lack comparisons with other advanced scalar or generative reward models, making the effectiveness of MELT less supported. It is critical to demonstrate that whether MELT could outperform these preference modeling paradigm.
  2. It is not clear how to select the set of feedback when dealing with an unseen task. From table 2, it seems that more combinations of feedbacks might degrade the performance (safety tasks).
Questions for authors
  1. What if the reward model output only the scalar not the reward distribution, i.e., can you explain the advantage of learning a reward distribution over predicting a scalar reward?
  2. Can you compare MELT with DPO using preference pairs constructed from edits and ratings, or explain the advantages of MELT over DPO in terms of policy model's performance?
1 comment
AuthorsOpenReview ↗

Response to Reviewer iVcT

Thank you for taking the time to review our work and your thoughtful questions. We are glad you found our approach practical and its technical contribution and policy-optimization performance convincing. We address your remaining concerns and questions below.

W1. Comparison to other reward models We do not compare against other scalar RM variants or generative/LLM-judge reward models. The comparison to our baseline, a scalar RM trained on pairwise preferences, lets us separate (1) the effect of the token-level formulation from (2) the benefit of additional feedback types, while holding backbone and data scale constant.

Comparisons with open SOTA RMs would involve differences in training-data scale and curation, often alongside differences in backbone size and architecture. Generative reward models also use a different modeling approach. For example, Skywork-Reward-V2-Llama-3.2-3B (Liu et al., 2025) shares our architecture, parameter count and a preference likelihood, but is trained on 26M curated preference pairs, approximately 460x our math and 650x our safety preference data. MELT is complementary to such scaling: it adds further feedback types to whatever preference data is available. To also cover other standard preference-based post-training pipelines, we report results for SFT->DPO and SFT->PPO (Scalar P) in our answer to Q2.

W2. Feedback set selection Our new results show that adding feedback types is not harmful by itself; we attribute the smaller gains of some combinations to differences between feedback sources. Under best-of-k selection in math, P+D, P+S, and P+D+S all select better responses than MELT P on all four benchmarks, and the three-type model is best on three of them. An excerpt is in our response to Reviewer 3sLN (W1b); full results are in Table 3 and App. B.1.
To study the feedback types with realistic data at scale, we drew them from established public datasets. Our safety edits, for instance, come from CAI/HH-style revisions, while preferences and demonstrations come from Nemotron-Safety. These datasets differ in their prompts, response generators, and annotators' notions of quality, so some sources may provide partly conflicting signals. This is consistent with prior work showing that the value of additional feedback depends on its informativeness relative to existing supervision (Baur et al., 2026; Metz et al., 2025). Collecting multiple feedback types for the same responses would help isolate the effect of feedback format; we see this as valuable future work.

Q1. Advantages of distributional rewards The reward distribution enables MELT's Bayesian view of multi-feedback learning, where each feedback type informs the reward through its likelihood, balanced against the prior and the TD-consistency term. With a point estimate, we would have to weight each feedback type's contribution by hand. More generally, there are many interesting potential advantages of learning a reward distribution, many of which are open research directions. For example, one could optimize for lower-tail performance instead of the average reward, so that severe failures become less likely, or use reward uncertainty to decide which completions need more samples or verification. Concretely, in the paper we use the distributional rewards for PPO, where we sample rewards from the distribution instead of using the mean. We believe that this may explain why, under increasing policy drift (Fig. 3b–c), the scalar reward model’s return keeps rising while GSM8K accuracy declines, whereas preference-only MELT maintains its accuracy.

Q2. Comparison with DPO. Using the same datasets, we compare SFT on demonstrations followed by DPO on preferences against PPO with MELT's jointly learned P+D reward model. At PPO step 800, MELT P+D achieves higher accuracy than SFT->DPO on GSM8K (44.3 ± 0.4% vs. 40.9 ± 0.4%) and MMLU-Pro Math (23.8 ± 0.3% vs. 20.6 ± 0.7%), with similar performance on MATH500 (14.2 ± 0.3% vs. 14.0 ± 0.8%). Results are mean ± SE over four runs; the math SFT and DPO checkpoints are selected by validation-set early stopping. These comparisons are reported in Sec. 5.4, Fig. 4.

This does not evaluate DPO on pairs constructed from edits or ratings. Such conversions are sometimes possible, but they lose information:

  • Edits: Original and revised responses can form a DPO pair, but whole-response DPO does not supervise which tokens changed. MELT uses the edit alignment as local reward evidence and the inserted or substituted tokens as corrective demonstrations.
  • Ratings: Positive and negative responses to the same prompt can form pairs, but MELT can also use individually rated responses without a same-prompt counterpart.
  • Stops: An interruption provides no preferred alternative. MELT directly models the stopping position and the absence of earlier intervention.

A conversion therefore optimizes a different objective than the original feedback likelihood.

Reviewer_3sLNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MELT, which learns a token-level reward model jointly from preferences, demonstrations, ratings, edits, and stops. Each feedback type enters through its own likelihood over a shared latent reward, the posterior is approximated by a Gaussian reward head on a pretrained transformer, and an action-value head initialized from the LM head is tied to the rewards through a TD-consistency term, following AVRIL/MAVRL.

On a Llama 3.2 3B backbone, combining feedback types improves error localization on perturbed GSM8K solutions, raises RewardBench 1 accuracy over preference-only and scalar reward models in math and safety, and gives higher and more drift-robust GSM8K accuracy under PPO. On RewardBench 2, gains appear in safety but not in math.

Critical strengths
  1. A clean, extensible formulation with two new feedback likelihoods. Treating every feedback type as an observation of one latent token-level reward, each with its own likelihood, is an elegant way to combine supervision without hand-weighted losses. The edit likelihood and the hazard-guided stop likelihood are natural and new for language models, as far as I know. The paper also addresses the practical obstacle of computing Q* for an LLM by initializing the action-value head from the LM head.

  2. Careful, honest and transparent experimental practice. Results are averaged over multiple seeds with standard errors, the PPO comparison uses several runs per reward model, and the appendix includes a matched-data control, ablations of the main design choices, and a thorough decontamination audit against the evaluation benchmarks. The paper also reports results that go against its own story, such as the lack of gains on RB2 math and the cases where adding a feedback type hurts, rather than leaving them out

Critical weaknesses
  1. The significant RB1 gains may come come from a benchmark shortcut (RB2 results is bad may confirm this suspicion).

A reward benchmark should only differ between good and bad responses in quality. RB1 math subset does not: its good responses are human-written reference solutions and its bad ones are GPT-4 solutions containing errors, so the two also differ in writing style. A reward model can score highly there by recognizing style alone, which is reward hacking [1]. MELT's math training data teaches exactly that shortcut. Demonstrations are human-written reference solutions; stops are PRM800K GPT-4 solutions truncated at the first error. A model trained on D and S learns "reference style = good, GPT-4 style = bad," which is what RB1 math rewards. Note that RB2 Math removes the shortcut: all four candidates are model-generated, and the model must pick the correct one. The results split exactly as a style shortcut predicts. D+S, which never sees a preference, scores 94.3% on RB1 but 30.8% on RB2. S alone falls from 66.1% to 35.2%. And on RB2 math, no MELT configuration beats the scalar reward model, as Section 5.3 shown.

Safety also shows this problematic suspicion. MELT-P and the scalar model share the same preference data, backbone, and objective, yet MELT-P is 11.8 points better on RB1 safety and no better on RB2. The one structural difference is that MELT scores a response by summing token rewards, which grows with length, and RB1 safety subsets have known length biases. Without a length-controlled analysis and an explanation for the weak scalar baseline, the "+17.7 over the scalar model" claim does not hold either.

  1. The paper's central claim is not properly tested, and the novelty is limited.

The introduction argues that sending each feedback type to a separate training stage (demonstrations to SFT, preferences to a reward model) wastes information, and that one reward model trained on all feedback does better. This is never tested. In the PPO experiment, the scalar baseline never sees the 57k demonstrations, while MELT P+D does, so P+D's gain could simply come from having more data. The obvious control, SFT on the same demonstrations followed by PPO with the scalar reward model, is missing.

The novelty is limited. [2] already formulates reward learning from multiple feedback types as inference over a shared latent reward with one likelihood per type, and already covers demonstrations, comparisons, ratings, and stops. [3] already applies Bayesian IRL with token-level rewards to LLM alignment, and is cited but not compared against. [4] already study reward learning from six feedback types, including corrections, which the paper does not cite. What MELT adds is the LLM-scale implementation and the edit likelihood, which is an incremental step, and the paper does not say clearly how little of the framework is new.

One more ablation is missing: for error localization, a language model already assigns low probability to a wrong number, so looking for the biggest log-probability drop under the base model might find errors as well as MELT does. This is especially plausible because MELT's action-value head starts from the LM head and is trained with next-token cross-entropy, so its rewards largely reflect how surprising each token is. The paper should report this baseline under the same setting.

REFERENCES:

[1] Kim, Sunghwan, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. "Evaluating robustness of reward models for mathematical reasoning." arXiv preprint arXiv:2410.01729 (2024).

[2] Baur, Raphaël, Yannick Metz, Maria Gkoulta, Mennatallah El-Assady, Giorgia Ramponi, and Thomas Kleine Buening. "MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference." arXiv preprint arXiv:2602.15206 (2026).

[3] Cai, Yuang, Yuyu Yuan, Jinsheng Shi, and Qinhong Lin. "Approximated variational bayesian inverse reinforcement learning for large language model alignment." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 22, pp. 23505-23513. 2025.

[4] Metz, Yannick, Andras Geiszl, Raphaël Baur, and Mennatallah El-Assady. "Reward learning from multiple feedback types." In International Conference on Learning Representations, vol. 2025, pp. 45154-45204. 2025.

Questions for authors

See weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3sLN

Thank you for the careful reading and constructive review. We are glad you appreciated the unified formulation, the edit and stop likelihoods, and our experimental practice.

W1a. Length bias (RB1 Safety). MELT still outperforms the scalar baseline significantly across feedback combinations when controlling for length in safety. To control for length, we split each subset by whether the correct response is shorter or longer than the incorrect one and report balanced accuracy (unweighted mean of both strata). A constant negative token reward would score ~60% on RB1 but ~50% under this metric. Full tables, including math, are in App. B.3

RB1 SafetyRaw Acc.Correct shorter (n=440)Correct longer (n=295)Balanced Acc.
Scalar (P)60.053.469.861.6
MELT P71.877.563.670.5
MELT P+D77.784.867.376.0
MELT P+E74.569.282.575.8
MELT P+R73.169.778.073.9

The length effect has no consistent direction: P and P+D do better on “Correct shorter”, P+E and P+R on “Correct longer”. One clarification on the comparison of Scalar (P) and MELT P: They share data and backbone but not the objective. MELT adds a KL term that acts as a regularizer; summation of the rewards across tokens is thus not the only difference.

W1b. Style shortcut (RB1 Math). The style shortcut likely explains the RB1 score of D+S, but this does not generalize: Under best-of-k, where all candidates come from the same source, P+D and P+D+S still select better responses than MELT P or the scalar model. We sample 16 responses per problem from the SFT policy. Each reward model selects its highest-scoring response; we report its accuracy (mean of 5 seeds). The full table is in App. B.1.

BoN@16GSM8KMATH500MMLUMMLU-Pro
SFT Policy30.48.133.619.0
Scalar (P)41.410.140.122.0
MELT P38.98.938.020.9
MELT P+D43.212.243.222.4
MELT P+S42.79.440.121.9
MELT D+S27.79.731.019.2
MELT P+D+S45.110.943.523.7

P+D and P+D+S exceed the scalar model on all four benchmarks; 2/8 differences are within noise (P+D on MMLU-Pro, P+D+S on MATH500). MELT P sits slightly below the scalar model, which supports that the gains primarily stem from combining feedback, not from the token-level formulation. Under best-of-k, D+S falls below the SFT Policy on GSM8K and MMLU, which is consistent with its RB1 Math score reflecting the style shortcut. Best-of-k is now part of the main text (Table 3), and we removed the RB1 Math gains from the abstract.

As suggested, we also evaluated RewardMATH (Kim et al., 2024). Performance is limited for all models: the scalar baseline scores 9.3% (random ranking: 10.0%), and the strongest result is P+D with 12.0% accuracy and 32.4 MRR. The full table is in App. B.2.

W2. Single-stage vs. multi-stage. We added the requested SFT → scalar-RM PPO baseline (Fig. 4), with the same data and initial checkpoint as MELT. At PPO step 800, MELT P+D achieves higher accuracy on GSM8K (44.3 ± 0.4% vs. 35.7 ± 0.2%) and MMLU-Pro Math (23.8 ± 0.3% vs. 22.4 ± 0.4%), with similar performance on MATH500 (14.2 ± 0.3% vs. 14.4 ± 0.4%). Results are mean ± SE over four runs; the baseline’s SFT checkpoint is selected using early stopping on the validation set.

W3. Related work and novelty. Thank you for the pointer, we now cite Metz et al. (2025) (App. A). AVA (Cai et al., 2025) is the closest prior work for language models. MELT differs in three main ways: (i) AVA trains each model on a single feedback type (Algorithm 1: demonstrations or preferences, never both). (ii) AVA does not model each feedback type through its own likelihood over a shared reward. Instead, it reduces preferences to positive and negative demonstrations. (iii) MELT adds likelihoods for edits, ratings and stops. AVA has no public implementation, and its objective differs enough from ours that we could not reproduce it faithfully. MAVRL is evaluated on tabular grids and simple control environments. MELT scales Bayesian reward inference to a 3B-parameter language model, adds an edit likelihood, and adapts the rating and stop likelihoods to token-level generation.

W4. Error localization. Section 5.2 does not aim to show that MELT beats a language model's log-probabilities at finding errors; it tests whether the rewards carry token-level information and, most importantly, whether combining feedback improves it. The baseline you suggest reaches 52.6 / 70.2 (Hit@1/5) under the LM initialization, above our reward model (P+D, 33.4 / 54.7). The rewards do not simply copy token surprisal: for a given model, the reward head μθ\mu_\theta and the action-value head QϕQ_\phi localize differently (D: 16.0 vs. 47.4; P+D: 33.4 vs. 44.6 Hit@1). We added the log-prob and QϕQ_\phi rows to Table 1.

I thank the author for the rebuttal. However, many of my concerns remain:

W1a. Author has not show me the result on RB2. Afaik, RB1 is a very problematic benchmark with the discussed limitation. Also, result the author gave me also shows that there's a very high variance between shorter and longer category, which confirms that length plays a significant role in the evaluation results. This may be a problem with the method or the RB1 benchmark, either way it shows that the current results is not stable, at least with RB1.

W1b. I appreciate the additional experiments, but I'm afraid that it seems unrelated and insufficient to my raised concern: my problem was with benchmarking the reward model, not how good they are for post-training like the authors shown. Also, it' very unclear how each reward type contributes to the learning, since the results (P+D+S or other combinations) are not stable across benchmark at all.

W2. Shouldn't the result at Step 0 be identical for the yellow line and red line? why is there a difference in their performance? Also, did you train SFT and SFT+DPO models in the same budget with PPO methods? I did not find any specification about this experiments to assess the fairness.

W3. Partly resolved. However, as I inspect the main results again, I do not really see the benefits of E, R and S. In many cases, using only P and D is enough, and adding more reward types could even hurt the performance.

W4 My point was that using such error localized method as a reward for training, would it beat MELT? If it does, that weakens the empirical findings of MELT.

I keep my score for now.

AuthorsOpenReview ↗

Response to Reviewer 3sLN Part I: W1a + W1b

W1a.

Author has not show me the result on RB2

Thank you for following up. Following your concern about length, we now also report RB2 with length control (App. B.3).

RB2 presents one preferred and three rejected responses per item. To control for length, we split items by whether the preferred response is among the two shortest or the two longest of the four candidates, and report balanced accuracy as the unweighted mean of the two halves. Random selection scores 25% in each half, and always choosing the shortest or the longest response scores about 30% and 23%, respectively.

RB2 SafetyRaw Acc.Correct in 2 shortest (n=236)Correct in 2 longest (n=214)Balanced Acc.
Scalar (P)50.147.553.050.3
MELT P51.950.453.451.9
MELT P+D55.057.752.154.9
MELT P+E56.552.560.756.6
MELT P+R62.254.970.762.8
MELT P+D+R63.058.168.963.5
RB2 MathRaw Acc.Correct in 2 shortest (n=98)Correct in 2 longest (n=85)Balanced Acc.
Scalar (P)54.950.660.255.4
MELT P53.345.563.154.3
MELT P+D55.150.859.555.2
MELT P+S54.646.364.555.4
MELT P+D+S51.839.266.853.0

In safety, the gains from combining feedback types remain under this control: P+R and P+D+R reach 62.8% and 63.5% balanced accuracy, compared with 51.9% for MELT P and 50.3% for the scalar model. They also outperform the scalar model when the preferred response is among the two shortest (54.9% and 58.1% vs. 47.5%), so their advantage does not come from preferring longer responses alone. In mathematics, no MELT configuration improves on the scalar model on RB2, with or without length control (53.0–55.4% vs. 55.4% balanced); we already report this in Section 5.3.

[…] result the author gave me also shows that there's a very high variance between shorter and longer category […]

We assume “variance” refers to the accuracy gap between the two length strata on RB1 safety (Table 7). We agree that this gap exists for every model, including the scalar baseline (16.4 points, vs. 8.3 to 17.5 points for the MELT variants in our previous response). This is why we report balanced accuracy: A heuristic that relies on length alone would only score about 50%. The scalar model reaches 61.6% and MELT 70.5% to 76.0%, so the advantage is not explained by a length exploit.

W1b.

[...] my problem was with benchmarking the reward model, not how good they are for post-training like the authors shown.

To address the concern about potential style shortcuts in RB1 Math from your original review, we introduced best-of-k as an additional reward-model evaluation. Importantly, best-of-k evaluates reward-model performance directly on standard mathematical reasoning benchmarks and is not a post-training experiment. The generating policy remains fixed, no policy optimization takes place, and all reward models rank the same set of sampled responses. The reported accuracies therefore reflect each reward model’s ability to select correct responses. This setup addresses the specific style shortcut identified in the original review, since correct and incorrect responses originate from the same generating policy rather than different sources. Both P+D and P+D+S achieve higher mean accuracy than the scalar baseline across all four benchmarks. These results provide independent evidence for the benefits of combining feedback beyond the potentially confounded RB1 evaluation.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period is coming to an end, I kindly encourage you to review and respond to the authors' rebuttal as soon as possible.

Thank you very much for your time and valuable contributions to the review process.

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.