- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MELT, a method for learning from different kinds of feedback including demonstrations, preferences, ratings, edits, and stops. It models each feedback through a feedback-specific likelihood and uses Bayesian inference to learn a shared distribution over token-level rewards. Its core idea is that each type of feedback provides different clues about what makes a response good or bad. The probabilistic reward model delivers rewards with uncertainty estimates, which are then used to optimize the policy. Experiments on mathematical reasoning and safety show that combining feedback types improves response ranking and error localization
Critical strengths
MELT combines demonstrations, preferences, edits, ratings, and stops through feedback-specific likelihoods. This allows different signals to inform one shared reward model while retaining their properties token-level rewards support error localization. The model can identify where a response goes wrong In mathematical reasoning, the paper holds the total number of feedback examples fixed. And provides evidence that complementary feedback helps beyond additional data alone The experiments examine different feedback combinations and report uncertainty/variations across runs
Critical weaknesses
It is unclear how much improvement comes from MELT itself. The scalar reward model uses only preferences, while MELT’s strongest results also use demonstrations and other feedback Adding more feedback types does not always help. MELT shows large gains on RewardBench math, but no clear improvement over the scalar reward model on RewardBench 2 math. In safety, combining all four feedback types performs worse than some combinations with fewer types The paper predicts a reward distribution for each token and shows its uncertainty, but does not establish whether these estimates reliably identify inaccurate predictions.
Questions for authors
does MELT outperform SFT followed by preference-based optimization? using the same demonstrations and preferences? What explains the limited gains on RewardBench 2 math and the weaker performance of the four-modality model in safety? Do higher predicted uncertainties correspond to less reliable reward predictions? How is the assumption that the observed successor action is near-optimal justified for trajectories containing errors?
Response to Reviewer ejux
Thank you for the constructive review. We appreciate your positive assessment of MELT’s formulation and experimental analysis. Your questions motivated additional comparisons with SFT→DPO, broader response-ranking evaluations, and an ablation of the TD target, which we discuss below.
W1: Uneven gains across benchmarks and feedback combinations.
RB2 Math*.* To assess whether the gains extend beyond RB1, we added best-of-16 evaluations on GSM8K, MATH500, MMLU, and MMLU-Pro, where all reward models rank the same candidates sampled from a shared policy. Each reward model selects its highest-scoring response; we report its accuracy (mean of 5 seeds):
MELT P+D and P+D+S outperform both preference-only MELT and the scalar model in mean accuracy on all four benchmarks. Preference-only MELT remains below the scalar model on all four benchmarks, supporting that these gains primarily come from combining feedback rather than the token-level formulation alone. These results are reported in Table 3, with all feedback configurations and standard errors in App. B.1.
Four-modality safety model. This model still exceeds preference-only MELT and the scalar model on RB1, and matches the scalar model on RB2. Its smaller gains relative to some combinations may reflect differences between feedback sources: edits come from CAI/HH-style revisions, while preferences and demonstrations come from Nemotron-Safety, with different prompts, response generators, and annotation criteria. Additional feedback may therefore offer limited benefit when its signals conflict with existing supervision. This is consistent with prior findings on feedback informativeness (Baur et al., 2026; Metz et al., 2025). Collecting multiple feedback types for the same responses would help isolate the effect of feedback format and we see this as a valuable direction for future work.
W2. Reliability of uncertainty estimates. Our experiments focus on the rewards learned through probabilistic multi-feedback inference and their usefulness for ranking and policy optimization. The posterior variance contributes to this process through TD-consistency weighting (Eq. 3) and reward sampling during PPO. Its usefulness as a separate reliability signal remains to be evaluated, as stated in the Sec. 6 of the original version of the paper.
Q1. Comparison with SFT followed by preference-based optimization. We ran an SFT->DPO baseline using the same demonstration and preference data and the same initial checkpoint as MELT. SFT->DPO uses demonstrations for additional math-specific SFT, while MELT uses them jointly with preferences to learn the reward model and starts PPO from the initial checkpoint. SFT and DPO checkpoints are selected by early stopping on the validation set; for MELT PPO, we report the final checkpoint (step 800).
Mean accuracy ± standard error over four runs. MELT achieves higher accuracy on GSM8K and MMLU-Pro Math, with similar performance on MATH500. These results support using demonstrations to inform reward learning alongside preferences, beyond their conventional use for SFT. These comparisons are in Sec. 5.4 (Fig. 4).
Q2. Applying TD consistency to all trajectories. Our ablations support the observed-action bootstrap as an effective practical approximation, with no consistent improvement from switching to a greedy maximum over the successor state. Although these continuations need not be optimal, the term can still provide a useful learning signal. We tested this approximation by replacing the observed successor value with $\max_{a'} Q_\phi(s_{t+1}, a')$, which drops the assumption that the recorded next action is near-optimal.
Entries report RewardBench 1 / RewardBench 2 accuracy, averaged over five seeds. Neither variant has a consistent advantage. Applying TD to demonstrations only gave worse results (RB2 Math: 50.5 vs. 55.1 for P+D), suggesting that TD on other feedback types' trajectories provides useful supervision in this setting. These results and a discussion of the assumption are in App. B.5 (Table 9).