- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PrefMind, an online dual-agent RL framework for multimodal preference alignment. A Generator and a Critic jointly learn preference judgment, correction, and response improvement through interactive feedback. Experiments on Qwen3-VL models at three scales demonstrate improvements in preference judgment and response generation.
Critical strengths
-
Important problem. The paper raises a meaningful question: whether explicitly learning human preferences can improve response generation beyond reward-score optimization.
-
Broad evaluation. The experiments cover multiple model scales and benchmarks, with generally clear presentation and implementation details.
Critical weaknesses
-
Limited novelty and insufficient validation of the core hypothesis. The method mainly combines existing techniques, including multi-agent discussion, preference judgment, and online RL. More importantly, it still relies on binary preference labels without additional supervision explaining the underlying preference criteria. It remains unclear whether the Generator genuinely learns the reasons behind human preferences or simply benefits from additional training signals.
-
Limited improvements and insufficiently controlled comparisons. PrefMind achieves only modest gains over the strongest baselines and does not outperform them at the 2B scale. Moreover, the dual-agent framework introduces substantial additional computation, but compute-matched comparisons are missing.
Questions for authors
The authors raise an important question in the abstract: "whether making preference understanding an internal capability of the response model can improve its generation." However, the evaluation still primarily relies on benchmark scores, similar to conventional RLHF methods, which does not convincingly demonstrate that preference understanding has become an internal capability of the Generator.
Have the authors considered rubric-based learning approaches, which seem more directly aligned with this objective? The research question is ambitious, but the information contained in binary preference labels is inherently limited. How can the proposed framework learn the underlying reasons for human preferences without richer supervision, and how can this capability be directly validated?
3 comments
Questions: Rubric-Based Approaches
We did consider this direction. Recent rubric-based methods (Rubrics as Rewards, OpenRubrics, Rubric-ARM, Auto-Rubric as Reward) first generate explicit evaluation criteria and then let a judge score the policy's outputs against them. In the multimodal setting, the closest method is Generative RLHF-V (GRM-RL in our tables), whose reward model infers the principles behind preferences before scoring. These methods improve the evaluator: the criteria are generated and used by other models, and the policy itself still receives only scores. They also require an additional rubric generator and per-criterion judging, and model-generated rubrics may deviate from human standards (RubricBench).
Our starting point is different: the Generator itself explicitly learns preference judgment, using only the existing binary labels and no rubric-generation pipeline. In the direct comparison with GRM-RL, PrefMind is better at all three scales. The two directions are complementary: rubrics could be used to verify the critiques directly, and learning from detailed rubrics might further improve the Generator’s understanding, which we leave as future work.
Questions: Evaluation Beyond Benchmark Scores
Besides generation benchmarks, we evaluate the Generator's standalone preference accuracy on held-out human-labeled pairs (Fig. 2), which measures preference judgment directly rather than through generation, together with the criterion experiment above.
W2: Computational Cost and Small-Model Performance
The full cost analysis is in our Response to All Reviewers. In short, on 4B PrefMind's total GPU-hours are 2.8× those of PPO and comparable to GRPO's (205.9 vs. 202.0), and PrefMind without replay uses fewer GPU-hours than GRPO (167.3) while still generating better than both. At 2B, the scalar-reward baselines gain 15–20 points on MM-Safety but their SIUO Effective score drops from 49.7 to 23–28, indicating broader refusals; PrefMind keeps it at 50.9. We agree that improving discussion-based training for small models remains future work.
We sincerely thank you again for your time and effort. We hope the additional analyses address your concerns, and we would be grateful if you would consider raising your score.
Followup on Rebuttal
Dear Reviewer:
Thank you again for taking the time to review our responses. As the rebuttal period will be ending soon, we would greatly appreciate it if you could kindly revisit our responses and let us know whether any concerns or questions remain unaddressed.
We would be more than happy to provide further clarification or additional explanations if anything is still unclear. If our responses and the additional experiments have adequately addressed your concerns, we would also sincerely appreciate your reconsidering the overall score in light of the new evidence. Thank you again for your time and consideration.
Novelty, Preference Understanding, and Additional Evidence
We thank you for the detailed review and for finding the problem important. Your questions on what the Generator actually learns have pushed us to add a direct test, which we describe below.
W1: Novelty and the Core Hypothesis
Our contribution is the question and the finding: explicitly training the response model's own preference judgment improves generation, whereas improving evaluator accuracy alone does not guarantee the same benefit. We support this with two pieces of evidence.
(i) Ablation. We removed the Generator's judgment loss while keeping everything else (its judgment is still used in the discussion with the Critic, and response-improvement training is unchanged). The Critic then becomes more accurate at all three scales, yet generation drops:
(ii) Evaluator accuracy vs. policy accuracy. The scalar reward model used by PPO and GRPO is the most accurate evaluator at 4B (79.8%), but the policies they train stay close to Base in their own preference accuracy (70.4% and 69.8% vs. 69.3%; revised Fig. 2), and their generation gains are smaller than PrefMind's, whose Generator reaches 72.6%.
Together, (i) and (ii) address whether the gains come simply from additional training signals. In (i), the interaction, the discussion, and the response-improvement training are all kept, so the Generator still receives the same extra signals; only its own judgment loss is removed, and generation drops at every scale. In (ii), a more accurate evaluator alone does not ensure that the policy becomes better at judging preferences or generating responses. These findings suggest that explicitly training the Generator's own preference judgment plays an important role beyond providing additional training signals or improving the external evaluator. We do not claim that this proves the Generator recovers the annotators' underlying reasons; we test that separately below with the criterion experiment.
W1 and Questions: Binary Labels and Whether the Generator Learns the Reasons
We acknowledge that our supervision is still the binary preference label, and that the critique text itself is not directly supervised. We do not add annotation; instead, the same label verifies several decisions in each interaction: the Generator's independent judgment, whether its feedback corrects or misleads the Critic, the Critic's revision, and whether the answer written after the discussion is preferred. A critique is therefore rewarded only through its verifiable effect on the other agent's judgment and on the generated answer.
To test directly whether the Generator has learned preference criteria, we added a 4B experiment. Each model writes one sentence stating the criterion that decides a comparison, without referring to either response or stating a choice. A frozen Qwen3-VL-4B-Instruct then judges the pair given the image, the question, the two responses, and this criterion. Accuracy is averaged over the seven preference test sets used in the paper.
Only the criteria written by the PrefMind Generator help the frozen judge, while those written by Base and the PPO policy slightly hurt it. We regard it as suggestive evidence that the Generator can state criteria that transfer to another model. In addition, when the discussion is removed from the generation prompt (everything else unchanged), generation drops at all three scales (by 0.13, 0.53, and 2.35 points at 2B, 4B, and 8B), providing further evidence that discussion content contributes to generation. The criterion experiment is reported in the appendix of the revised paper, and we revised the abstract to state our claim more precisely: the Generator is trained to understand preferences through explicit judgment and correction, rather than supervised with the reasons themselves.