acceptodds
Under review as a conference paper at ICLR 2027

Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation

Abstract

Multimodal preference alignment often follows a score-based RLHF pipeline: a reward model is first trained to learn human preferences, then used to score on-policy responses and optimize the response model's generation. This turns preference supervision into a scalar signal for generation: a model may learn to produce answers with higher preference scores, but is rarely trained explicitly to learn the reasons underlying human preferences. We therefore ask whether making preference understanding an internal capability of the response model can improve its generation. To investigate this question, we propose PrefMind, an online dual-agent reinforcement learning framework with two trainable agents: a response Generator and a preference Critic. Given a pair of candidate responses, the agents analyze and discuss which response better aligns with human preference. Human labels supervise their judgments and corrections. For pairs on which both the Generator and revised Critic correctly identify the human-preferred response, the Generator produces an improved answer, and the Critic compares it against the original preferred response to provide a generation reward. Both agents are jointly updated from interactions sampled with their current policies. Through this process, PrefMind trains the Generator to understand human preferences via explicit judgment and correction, and to translate this understanding into better responses. Experiments across three model scales show improvements in both standalone preference judgment and downstream response generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLM, RLHF

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GLjsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PrefMind, an online dual-agent RL framework for multimodal preference alignment. A Generator and a Critic jointly learn preference judgment, correction, and response improvement through interactive feedback. Experiments on Qwen3-VL models at three scales demonstrate improvements in preference judgment and response generation.

Critical strengths
  1. Important problem. The paper raises a meaningful question: whether explicitly learning human preferences can improve response generation beyond reward-score optimization.

  2. Broad evaluation. The experiments cover multiple model scales and benchmarks, with generally clear presentation and implementation details.

Critical weaknesses
  1. Limited novelty and insufficient validation of the core hypothesis. The method mainly combines existing techniques, including multi-agent discussion, preference judgment, and online RL. More importantly, it still relies on binary preference labels without additional supervision explaining the underlying preference criteria. It remains unclear whether the Generator genuinely learns the reasons behind human preferences or simply benefits from additional training signals.

  2. Limited improvements and insufficiently controlled comparisons. PrefMind achieves only modest gains over the strongest baselines and does not outperform them at the 2B scale. Moreover, the dual-agent framework introduces substantial additional computation, but compute-matched comparisons are missing.

Questions for authors

The authors raise an important question in the abstract: "whether making preference understanding an internal capability of the response model can improve its generation." However, the evaluation still primarily relies on benchmark scores, similar to conventional RLHF methods, which does not convincingly demonstrate that preference understanding has become an internal capability of the Generator.

Have the authors considered rubric-based learning approaches, which seem more directly aligned with this objective? The research question is ambitious, but the information contained in binary preference labels is inherently limited. How can the proposed framework learn the underlying reasons for human preferences without richer supervision, and how can this capability be directly validated?

3 comments
AuthorsOpenReview ↗

Novelty, Preference Understanding, and Additional Evidence

We thank you for the detailed review and for finding the problem important. Your questions on what the Generator actually learns have pushed us to add a direct test, which we describe below.

W1: Novelty and the Core Hypothesis

Our contribution is the question and the finding: explicitly training the response model's own preference judgment improves generation, whereas improving evaluator accuracy alone does not guarantee the same benefit. We support this with two pieces of evidence.

(i) Ablation. We removed the Generator's judgment loss while keeping everything else (its judgment is still used in the discussion with the Critic, and response-improvement training is unchanged). The Critic then becomes more accurate at all three scales, yet generation drops:

Setting2B4B8B
Full PrefMind: Gen. avg.59.3373.3176.41
No Generator judgment loss: Gen. avg.59.0372.3773.35
No Generator judgment loss: Critic acc. (Δ)+1.67+0.50+6.86

(ii) Evaluator accuracy vs. policy accuracy. The scalar reward model used by PPO and GRPO is the most accurate evaluator at 4B (79.8%), but the policies they train stay close to Base in their own preference accuracy (70.4% and 69.8% vs. 69.3%; revised Fig. 2), and their generation gains are smaller than PrefMind's, whose Generator reaches 72.6%.

Together, (i) and (ii) address whether the gains come simply from additional training signals. In (i), the interaction, the discussion, and the response-improvement training are all kept, so the Generator still receives the same extra signals; only its own judgment loss is removed, and generation drops at every scale. In (ii), a more accurate evaluator alone does not ensure that the policy becomes better at judging preferences or generating responses. These findings suggest that explicitly training the Generator's own preference judgment plays an important role beyond providing additional training signals or improving the external evaluator. We do not claim that this proves the Generator recovers the annotators' underlying reasons; we test that separately below with the criterion experiment.

W1 and Questions: Binary Labels and Whether the Generator Learns the Reasons

We acknowledge that our supervision is still the binary preference label, and that the critique text itself is not directly supervised. We do not add annotation; instead, the same label verifies several decisions in each interaction: the Generator's independent judgment, whether its feedback corrects or misleads the Critic, the Critic's revision, and whether the answer written after the discussion is preferred. A critique is therefore rewarded only through its verifiable effect on the other agent's judgment and on the generated answer.

To test directly whether the Generator has learned preference criteria, we added a 4B experiment. Each model writes one sentence stating the criterion that decides a comparison, without referring to either response or stating a choice. A frozen Qwen3-VL-4B-Instruct then judges the pair given the image, the question, the two responses, and this criterion. Accuracy is averaged over the seven preference test sets used in the paper.

Criterion given to the frozen judgePreference acc. (%)
None69.3
Written by Base68.1
Written by the PPO policy67.9
Written by PrefMind Generator70.5

Only the criteria written by the PrefMind Generator help the frozen judge, while those written by Base and the PPO policy slightly hurt it. We regard it as suggestive evidence that the Generator can state criteria that transfer to another model. In addition, when the discussion is removed from the generation prompt (everything else unchanged), generation drops at all three scales (by 0.13, 0.53, and 2.35 points at 2B, 4B, and 8B), providing further evidence that discussion content contributes to generation. The criterion experiment is reported in the appendix of the revised paper, and we revised the abstract to state our claim more precisely: the Generator is trained to understand preferences through explicit judgment and correction, rather than supervised with the reasons themselves.

Questions: Rubric-Based Approaches

We did consider this direction. Recent rubric-based methods (Rubrics as Rewards, OpenRubrics, Rubric-ARM, Auto-Rubric as Reward) first generate explicit evaluation criteria and then let a judge score the policy's outputs against them. In the multimodal setting, the closest method is Generative RLHF-V (GRM-RL in our tables), whose reward model infers the principles behind preferences before scoring. These methods improve the evaluator: the criteria are generated and used by other models, and the policy itself still receives only scores. They also require an additional rubric generator and per-criterion judging, and model-generated rubrics may deviate from human standards (RubricBench).

Our starting point is different: the Generator itself explicitly learns preference judgment, using only the existing binary labels and no rubric-generation pipeline. In the direct comparison with GRM-RL, PrefMind is better at all three scales. The two directions are complementary: rubrics could be used to verify the critiques directly, and learning from detailed rubrics might further improve the Generator’s understanding, which we leave as future work.

Questions: Evaluation Beyond Benchmark Scores

Besides generation benchmarks, we evaluate the Generator's standalone preference accuracy on held-out human-labeled pairs (Fig. 2), which measures preference judgment directly rather than through generation, together with the criterion experiment above.

W2: Computational Cost and Small-Model Performance

The full cost analysis is in our Response to All Reviewers. In short, on 4B PrefMind's total GPU-hours are 2.8× those of PPO and comparable to GRPO's (205.9 vs. 202.0), and PrefMind without replay uses fewer GPU-hours than GRPO (167.3) while still generating better than both. At 2B, the scalar-reward baselines gain 15–20 points on MM-Safety but their SIUO Effective score drops from 49.7 to 23–28, indicating broader refusals; PrefMind keeps it at 50.9. We agree that improving discussion-based training for small models remains future work.

We sincerely thank you again for your time and effort. We hope the additional analyses address your concerns, and we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Followup on Rebuttal

Dear Reviewer:

Thank you again for taking the time to review our responses. As the rebuttal period will be ending soon, we would greatly appreciate it if you could kindly revisit our responses and let us know whether any concerns or questions remain unaddressed.

We would be more than happy to provide further clarification or additional explanations if anything is still unclear. If our responses and the additional experiments have adequately addressed your concerns, we would also sincerely appreciate your reconsidering the overall score in light of the new evidence. Thank you again for your time and consideration.

Reviewer_7XdVOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes PrefMind, which trains a response model to judge response quality, correct evaluations, and improve answers. It jointly trains a Generator and a Critic that discuss candidate responses, with their judgments checked against human preference labels. For examples that pass this check, the Generator produces an improved answer, and the Critic provides a reward. The goal is to translate preference judgment ability into better response generation. The authors evaluate the approach at three model scales through preference judgment and response generation benchmarks, along with ablation studies.

Critical strengths
  1. Training the response model to evaluate answers and correct judgments is a worthwhile direction. The method makes identifying better responses an explicit training task and evaluates this ability separately, providing a way to study the relationship between evaluation and generation abilities.

  2. The trained response model can be used independently, without further discussion between the two models at inference time. At 4B and 8B, it achieves the highest average generation scores among the compared methods, suggesting that the training procedure offers some practical benefit.

Critical weaknesses
  1. The method adds several training stages, but it is unclear whether the gains justify the cost. 2-model discussion, answer revision, and failed-case replay all require additional computation, while the average generation scores exceed PPO by only 1.3 and 1.7 points at 4B and 8B, and the 2B model does not lead. The paper reports neither training costs nor comparisons under matched compute budgets, making it difficult to assess the practical advantage over simpler methods.

  2. The experiments do not directly show how learning preference judgment helps generation. The ablations mainly examine task weights, replay, and single versus 2-model training, without separately testing the effect of removing the Generator’s judgment training or the discussion between the two models. The paper itself finds that more accurate judgments do not necessarily lead to better generation. The results therefore support the effectiveness of the overall procedure, but do not establish that the gains come from learning to understand preferences.

Questions for authors
  1. Could the authors provide training time and computational costs relative to baselines such as PPO? Have comparisons under similar compute budgets been explored?

  2. Have the authors tried retaining answer-revision training while removing either the Generator’s preference-judgment training or the discussion between the two models? It would be helpful to understand how much each component contributes to generation quality.

3 comments
AuthorsOpenReview ↗

Performance Gains and Component Ablations

We thank you for the thoughtful review and for considering this a worthwhile direction. Your questions on cost and on the role of each component have led us to add the analyses below.

W1: Performance Gains

The training cost and the comparison with PPO and GRPO are in our Response to All Reviewers (see Q1 below for the key numbers). Regarding the gains, PrefMind achieves the best result on 5 of 8 primary metrics at 4B and 6 of 8 at 8B, so the improvement is not driven by a single metric. At 2B it does not have the highest average, but it offers a more favorable safety–helpfulness trade-off: DPO, PPO, and GRPO raise MM-Safety by 15–20 points but reduce SIUO Effective from 49.7 to 23–28, suggesting over-refusal, whereas PrefMind keeps SIUO Effective at 50.9 and has the best SIUO Both, LLaVA-Wilder, and MM-Vet scores, albeit with smaller safety gains. This pattern is consistent with our core insight: the Generator benefits from learning to understand the preference itself, rather than only optimizing for a high score from an external reward.

W2: Relationship Between Preference Judgment and Generation

Regarding "better judgment does not necessarily lead to better generation", we have revised Fig. 2 to separate two quantities that the original figure conflated: the preference accuracy of the evaluator and that of the response model itself. PPO and GRPO are guided by the most accurate evaluator, yet their trained policies remain close to Base in their own preference accuracy at every scale, and their 4B generation gains are only +1.1 and -0.4. PrefMind Generator is more accurate itself and attains the largest generation gain (+2.43). The ablation in our response to Q2 points the same way: without the Generator's judgment loss, the Critic becomes more accurate at all three scales, yet generation degrades. What matters for generation is therefore the response model's own judgment ability rather than the accuracy of its evaluator. Judgment accuracy alone is also not sufficient: placing more weight on judgment (β=0.5\beta=0.5) raises accuracy but lowers the generation gain.

Q1: Computational Cost

On 4B (A100-80GB), PrefMind needs 205.9 GPU-hours versus 74.0 for PPO and 202.0 for GRPO with 8 responses per prompt; PrefMind without replay needs 167.3 GPU-hours and still generates better than both. Rollout tokens are 32.9M versus 28.85M for PPO and 70.4M for GRPO, because PrefMind's outputs are short (discussion outputs and improved answers average 67.8 and 121.7 tokens, versus 366.4 and 224.2 for PPO and GRPO answers). Total GPU-hours include evaluator training, and the extra cost is training-only: at inference the Generator answers alone, without discussion. The full table is in our Response to All Reviewers and has already been added to the appendix.

Q2: Component Ablations

We ran both variants, keeping answer-revision training in each:

  • (a) No Generator judgment loss: The Generator's judgment rollout is still used in the discussion, but its preference-judgment loss is removed.
  • (b) No discussion in the revision prompt: The discussion history is removed from the prompt of the final revision step, while the discussion during training is unchanged.
ScaleSettingGenerator pref.Critic pref.Gen. avg.
2BFull PrefMind64.4773.1559.33
No Generator judgment loss60.59 (-3.88)74.82 (+1.67)59.03 (-0.30)
No discussion in revision prompt61.94 (-2.53)74.34 (+1.19)59.20 (-0.13)
4BFull PrefMind72.6576.9273.31
No Generator judgment loss71.30 (-1.35)77.42 (+0.50)72.37 (-0.94)
No discussion in revision prompt72.84 (+0.19)78.12 (+1.20)72.78 (-0.53)
8BFull PrefMind72.8472.1676.41
No Generator judgment loss68.15 (-4.69)79.02 (+6.86)73.35 (-3.06)
No discussion in revision prompt75.04 (+2.20)70.24 (-1.92)74.06 (-2.35)

Generator judgment training. Removing the Generator's judgment loss consistently reduces both its preference accuracy and generation performance, despite improving the Critic's preference accuracy at every scale. Most notably, at 8B, generation drops by 3.06 points, falling below PPO (74.7). At 4B, the ablation eliminates 0.94 of PrefMind's 1.31-point advantage over PPO. These results demonstrate that improving the external evaluator alone cannot replace explicitly training the Generator to understand human preferences.

Discussion-informed revision. Removing discussion content from the revision prompt also decreases generation performance across all scales, with drops of 0.13, 0.53, and 2.35 points at 2B, 4B, and 8B, respectively. This indicates that the information exchanged during discussion contributes to the Generator's ability to revise its responses.

Together, these results provide direct ablation evidence that both the Generator's preference-judgment training and the use of discussion in answer revision contribute to generation quality, with judgment training showing the larger effect in these experiments.

We sincerely thank you again for your time and effort. We hope the cost comparison and the ablations you requested address your concerns, and we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow up on Rebuttal

Dear Reviewer:

Thank you again for taking the time to review our responses. As the rebuttal period will be ending soon, we would greatly appreciate it if you could kindly revisit our responses and let us know whether any concerns or questions remain unaddressed.

We would be more than happy to provide further clarification or additional explanations if anything is still unclear. If our responses and the additional experiments have adequately addressed your concerns, we would also sincerely appreciate your reconsidering the overall score in light of the new evidence. Thank you again for your time and consideration.

Reviewer_MzPzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PrefMind, an online reinforcement learning framework that jointly trains a multimodal response Generator and a preference Critic. The agents discuss and revise judgments on human-labeled response pairs. Correct Generator and revised Critic judgments gate response improvement, after which the Critic compares the new answer with the original preferred answer to reward the Generator. Both policies are updated with PPO, with optional failed-case replay. Experiments at 2B, 4B, and 8B evaluate standalone preference judgment and downstream response generation; the method improves over the corresponding base models and achieves the highest reported average generation score among the compared methods at 4B and 8B.

Critical strengths
  1. The paper evaluates both standalone preference judgment and downstream generation across three model scales and a range of safety, visual assistance, and integrated multimodal benchmarks. This is useful because better preference discrimination alone does not establish improved generation.

  2. The reported generation results are promising at 4B and 8B: PrefMind achieves the highest average among the compared methods, with improvements over Base of approximately 2.4 and 3.5 points, respectively. The separate reporting of safety and effectiveness also helps expose trade-offs that a safety score alone could obscure.

Critical weaknesses
  1. The incremental methodological contribution over prior work is not sufficiently clear. Prior work already studies generative preference modeling (Generative RLHF-V), transfer from preference judgment to generation (LLaVA-Critic-R1), and collaborative reinforcement learning for correction and persuasion (MAPoRL). PrefMind combines closely related components through bidirectional feedback, human-label-based generation gating, and joint optimization, but the current experiments do not sufficiently isolate what this combination adds. The reported ablations vary the judgment–generation balance, replay, and parameter sharing; they do not directly establish the necessity of bidirectional correction or label-verified gating. More explicit positioning against the closest methods and targeted ablations are needed to substantiate the claimed methodological advance.

  2. The fairness of the baseline comparison is not sufficiently established. Matching training data and individual backbone size does not establish a matched training budget: PrefMind updates two separate policies, collects multiple interaction stages, and replays failed examples. The paper does not provide enough information about total rollout tokens, optimization work, or GPU-hours across methods to determine whether its gains come from the proposed interaction mechanism or additional computation and repeated supervision. Please report the full training costs, including evaluator training where applicable, and compare against strong baselines under a matched total compute or rollout-token budget. This is important for assessing the practical advantage of the method.

Questions for authors
  1. What is the incremental benefit of bidirectional preference discussion over simpler alternatives? In particular, since the Generator receives the human-preferred response during the improvement stage, can you compare against directly refining that same response with the same Critic reward, without the preceding discussion? Please control the selected generation examples and training budget where possible, and clarify the separate contributions of correction and label-based gating.

  2. What are the total rollout-token counts, optimization costs, and GPU-hours for PrefMind and the strongest baselines, including reward-model or Critic training where applicable? Does the advantage remain under a matched total training budget? These results would materially affect my assessment of the method's practical benefit.

8 comments
AuthorsOpenReview ↗

Novelty Declaration and Ablation Study

We thank you for the careful review and for recognizing that evaluating both standalone preference judgment and downstream generation across three model scales is useful, and that separately reporting safety and effectiveness helps expose trade-offs. Your questions on the contribution over prior work and on matched compute have helped us strengthen the paper, and we respond to them below.

W1: Contribution Over Prior Work

PrefMind's contribution is to have the Generator itself learn to judge human preferences and use that judgment to improve its own responses, with a co-trained Critic providing the reward. Relative to the cited works:

  • In Generative RLHF-V, generative preference judgment lives in the reward model and the response model only receives its scalar-based scores. In PrefMind, the Generator itself is trained to judge and correct human preferences, and to apply this judgment when improving its own responses.

  • LLaVA-Critic-R1 trains models on critic data and demonstrates transfer gains in general reasoning capabilities; it does not directly examine whether preference understanding enables a model to generate better preference-aligned responses. PrefMind makes this link an explicit training objective: the Generator's own judgment is verified against human labels, discussed with the Critic, and then used to write an improved answer, whose quality is rewarded by the Critic.

  • MAPoRL, like most multi-agent LLM work, trains agents as symmetric collaborators that jointly solve a task such as math or logical reasoning, and they collaborate again at inference. In PrefMind, the Critic is not a co-solver but a learned stand-in for the reward model: it judges human preferences and scores the Generator's new response. Supervision comes from human preference labels rather than task correctness, and the Generator produces responses on its own after training.

Q1: Additional Ablation Studies

Thank you for suggesting these targeted ablations. We ran four additional 4B experiments, all retaining response-improvement training:

  • (a) No Generator judgment loss: The Generator's judgment rollout is kept for the discussion, but its preference-judgment loss is removed.
  • (b) No discussion in improvement prompt: The discussion history is removed from the prompt of the final generation step.
  • (c) Without Critic revision: The Critic's feedback-conditioned revision is removed, and the generation gate is applied to the Generator's judgment and the Critic's initial judgment.
  • (d) Random generation gate: Discussion and revision are retained, but interactions are selected for generation at random, matching the full model's number of generation examples at each training step.
SettingGenerator Acc.Critic Acc.Generation Avg.
Full PrefMind72.6576.9273.31
(a) No Generator judgment loss71.30 (-1.35)77.42 (+0.50)72.37 (-0.94)
(b) No discussion in improvement prompt72.84 (+0.19)78.12 (+1.20)72.78 (-0.53)
(c) Without Critic revision72.90 (+0.25)77.80 (+0.88)72.18 (-1.13)
(d) Random generation gate72.50 (-0.15)76.80 (-0.12)72.61 (-0.70)

What the discussion adds. Across all four ablations, standalone preference accuracy is largely preserved (the Critic's even increases in three of them), yet generation drops in every case. The benefit of the discussion is therefore not a more accurate judge. A binary label can supervise the A/B choice; the discussion is what makes the content of the Generator's analysis consequential. The analysis must help the Critic correct its judgment to earn the correction reward (removed in (c)), must be verified against the human label before it is used (removed in (d)), and is then used to improve the response (removed in (b)). In these three variants the Generator's accuracy changes by at most 0.25 points, but generation drops by 0.53–1.13 points.

Refining without the preceding discussion. In PrefMind the Critic is itself trained through the discussion, so removing the discussion entirely would also change the Critic that provides the reward. We therefore did not run this exact control. Variants (a) and (b) instead keep the Critic's training and remove the discussion's two roles for the Generator: learning from it (a) and using it during improvement (b). In (a) the Critic becomes more accurate (+0.50) while generation drops by 0.94 points, so a better external reward does not substitute for the Generator's own judgment training.

Correction and gating. Removing Critic revision (c) costs 1.13 points; replacing the label-based gate with random selection (d) costs 0.70 points. The latter gap is modest partly because the gate already accepts 70.29% of interactions.

Q2: Computational Cost

The full cost table is in our Response to All Reviewers. In short, on 4B PrefMind generates 32.9M rollout tokens (1.14× PPO, 2.1× fewer than GRPO) and uses 205.9 GPU-hours (2.8× PPO, comparable to GRPO's 202.0), including Critic training. PrefMind without replay uses 167.3 GPU-hours, fewer than GRPO, and still generates better than both PPO and GRPO (Fig. 2(d)). The additional cost is training-only: at inference the Generator answers alone, without discussion.

We sincerely thank you again for your time and effort. We hope the additional ablations and the cost comparison you requested address your concerns, and we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

One extra ablation added for Q1

Single-agent self-critique baseline. We additionally trained a single 4B Generator that plays both roles: it judges the response pair (rewarded by the human label), revises its judgment given the label, generates an improved response, and then judges its own new response against the human-preferred one. The judgment:generation loss ratio (1:3), replay, and gating match full PrefMind.

SettingPreference Acc.Generation Avg.
Base69.370.9
Single-agent self-critique75.271.8
Full PrefMind72.65 (Generator) / 76.92 (Critic)73.31

The single agent is a better judge than our Generator (+2.6) yet a worse generator (−1.5), gaining only 0.8 over Base. A plausible explanation is self-reward hacking: when the same policy produces a response and scores it, it can raise its reward by exploiting its own judging biases rather than by improving the response. This is well documented for single-model self-evaluation and self-refinement (Xu et al., 2024; Panickssery et al.).

Together with ablation (a), this gives a two-sided picture: an external reward without the Generator's own judgment training is insufficient (ablation (a)), and the Generator's own judgment without an independent scorer is also insufficient (this experiment). Human labels cover only the original response pair, not the newly generated on-policy response, so an independent Critic is needed to score it. This is why PrefMind is a dual-agent framework.

We sincerely thanks for your recommendation and hope you could consider raising your score.

References

  • Xu et al., 2024. Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement. ACL 2024.
  • Panickssery et al., 2024. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to All Reviewers

We sincerely thank all three reviewers for the time and care they put into reading our paper and writing detailed reviews. We are encouraged by their recognition of the importance of the problem and the research direction (7XdV, GLjs), the insights on the relationship between preference judgment and generation (MzPz, 7XdV), the clear presentation (MzPz, GLjs). Their comments have helped us sharpen both the claims and the evidence. Computational cost was a concern shared by all three reviewers, so we answer it once here.

Computational Cost Analysis.

We report the full training cost of PrefMind and of the standard baselines, PPO and GRPO. Costs of the 4B runs on A100-80GB GPUs are below. Rollout tokens are all tokens generated during training; Optimization is the GPU-hours spent on parameter updates. Total GPU-hours include evaluator training: PPO and GRPO share one separately trained scalar reward model, whereas the PrefMind Critic is trained jointly with the Generator, so its cost is already included in PrefMind's time. The same response length limit is applied to all methods.

MethodAvg. output (tokens)Rollout tokensOptim. GPU-hTotal GPU-h
PPO (2 epochs)366.428.85M65.1274.0
GRPO (8 responses/prompt)224.270.4M151.5202.0
PrefMind67.8 / 121.732.9M172.99205.9
PrefMind w/o replay--24.1M138.76167.3

PrefMind produces about 8.8 outputs per prompt visit (6 discussion judgments, 1.4 improved answers, 1.4 Critic comparisons), but they are short (discussion outputs and improved answers average 67.8 and 121.7 tokens). Its rollout tokens are 1.14× PPO and 2.1× fewer than GRPO; its total GPU-hours are 2.8× PPO and comparable to GRPO. PrefMind without replay uses fewer GPU-hours than GRPO (167.3 vs. 202.0) and still generates better than both PPO and GRPO (Fig. 2(d)). The extra cost is training-only: at inference the Generator answers alone, without discussion. We have also added this analysis to the Appendix.

Beyond the cost analysis, each reviewer raised different concerns about PrefMind. We address these point by point in the individual responses below, where we add targeted ablations and further analyses for each reviewer's specific questions.

Thank you again for your time and your valuable feedback.

AuthorsOpenReview ↗

Summary of Additional Results and Where They Are Addressed

Below we summarize our novelty again and the main additional experiments and analyses we added in the individual responses, organized by the concerns the reviewers raised.

What PrefMind is, and what is new. MLLM RLHF typically converts human preference labels into a scalar score from a reward model; the response model is optimized against that score and is never trained to judge preferences itself. PrefMind is an online dual-agent RL framework for MLLM alignment: a Generator and a Critic, both trainable and updated jointly. For each human-labeled pair they discuss which response is preferred, and the human label supervises every step. The Generator is rewarded when its analysis helps the Critic correct a wrong judgment and penalized when it misleads a correct one. Only label-verified discussions are allowed to proceed to generation: the Generator then writes an improved answer, which the co-trained Critic compares against the original preferred response to produce the generation reward.

Three design choices distinguish PrefMind from prior work: (1) the response model itself is trained as a preference judge inside the RLHF loop, rather than delegating judgment to a separate reward model; (2) bidirectional, label-verified feedback, where the Generator's analysis is rewarded for its effect on the Critic's judgment; and (3) label-gated generation with an online co-trained Critic. Our additional experiments test each of these choices.

Our central finding. Explicitly training the response model's own preference judgment improves its generation, whereas a more accurate external evaluator alone does not; and a single model playing both roles, without an independent scorer, does not reproduce the benefit. We do not claim that the Generator recovers annotators' underlying reasons; we examine what binary labels can and cannot support below.

ExperimentResultWhat we can conclude
Remove the Generator's judgment loss (2B/4B/8B), keeping data, discussion and improvement training unchangedGeneration drops at every scale (59.33→59.03, 73.31→72.37, 76.41→73.35) even though the Critic becomes more accurate (+1.67, +0.50, +6.86).The gain depends on training the Generator's own judgment. A more accurate Critic cannot substitute for it, so the gain is not merely from extra training signal.
Remove parts of the discussion (4B): discussion in the improvement prompt, Critic revision, label-based generation gateFull 73.31; without discussion in the prompt 72.78; without Critic revision 72.18; random gate 72.61. Standalone preference accuracy is largely preserved.The discussion content, the Critic's revision, and label-based selection each contribute to generation beyond the Generator's judgment accuracy.
Evaluator accuracy vs. the policy's own accuracy (4B, revised Fig. 2)The scalar RM behind PPO/GRPO is the most accurate evaluator (79.8%), but the policies stay near Base in their own accuracy (70.4%, 69.8% vs. 69.3%) and gain +1.1/−0.4 in generation. The PrefMind Generator reaches 72.6% and gains +2.43.Evaluator accuracy does not by itself make the policy a better judge or generator. What matters is the response model's own judgment ability.
Criterion transfer (4B): a frozen Qwen3-VL-4B judges pairs given a one-sentence criterion written by different modelsNo criterion 69.3; written by Base 68.1; by the PPO policy 67.9; by the PrefMind Generator 70.5.The Generator's analysis carries preference information that another model can use, which Base and PPO do not provide, which might suggest that trained generator itself's understanding towards preference is increased.

Overall. Taken together, these results support one conclusion: PrefMind's gains come from training the Generator itself to judge preferences and from using that judgment, verified against human labels and tested through the co-trained Critic, to improve its own responses. They do not come from a more accurate external evaluator or from extra training signal alone.

Manuscript changes. Following the reviewers' suggestions, we made the following improvements to the revised paper: The component ablations are in Sec. 4.5 and Appendix E; the direct-generation ablation is in Appendix G; the criterion experiment is in Appendix H; the cost analysis is in Appendix D. Fig. 2 now separates the evaluator's accuracy from the policy's own accuracy, and the abstract states the claim more precisely.

We sincerely thank all reviewers for their reviewing. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.