- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the validity of scoring rubrics in multimodal evaluation and introduces MRH-Bench to assess MLLMs’ ability to identify and repair erroneous rubrics. The authors further propose QAPO and train Qwen3-VL-8B through SFT followed by QAPO to obtain MRH-Guard. Experiments show that MRH-Guard outperforms existing baselines and approaches the performance of closed-source models.
Critical strengths
- The problem is practically meaningful. The paper investigates potential hallucinations in judging rubrics for multimodal evaluation, offering a useful perspective on improving the reliability of MLLM evaluation systems.
- The data construction and evaluation analyses are reasonably thorough. MRH-Bench is carefully designed, with generated samples verified by two independent models and three experts, and pre-adjudication agreement statistics reported. Appendix H additionally provides human calibration on 480 outputs and validation using a second judge on matched subsets. These experiments increase confidence in the automated evaluation results.
Critical weaknesses
- Potential mismatch between synthetic and real-world data. The benchmark constructs hallucinated rubrics by introducing specific errors into valid rubrics using an LLM. However, my core concern about this work is that the paper does not sufficiently characterize the manifestations, prevalence, or distribution of rubric hallucinations in real-world settings. Without analyzing naturally occurring rubric hallucinations and checking their consistency or similarity with the synthetic examples, the practical value of this work remains uncertain. The resulting benchmark may primarily measure performance on a narrowly defined, rule-based task for MLLM judges, with limited generalizability.
- The improvements from QAPO are limited, and their statistical reliability remains unclear. Although QAPO is well motivated, its gains over the GRPO baseline are modest. Moreover, the quantitative results for diagnostic classification and repair quality rely on LLM judges, raising concerns about reliability. Without reported training and inference variability or confidence intervals, it is difficult to establish whether the observed differences from GRPO are statistically reliable.
Questions for authors
- Can the authors characterize naturally occurring rubric hallucinations, including their prevalence and distribution across the six proposed categories? How closely do the synthetic errors in MRH-Bench resemble these real-world failures?
- Can the authors evaluate MRH-Guard on a separately collected set of naturally occurring erroneous rubrics to assess whether training on controlled synthetic injections generalizes beyond the benchmark’s construction procedure?
- Can the authors report results across multiple training seeds and paired confidence intervals for the main comparisons between QAPO and GRPO, particularly for classification Macro-F1, repair success, and downstream judge–human agreement?
If the authors convincingly address these concerns, I would consider raising my rating.