acceptodds
Withdrawn from ICLR 2027

Rubric Can Hallucinate Too: Diagnosing and Repairing Multimodal Rubric Hallucination

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As multimodal large language models (MLLMs) are deployed in increasingly open-ended settings, scalar correctness and holistic preference scores provide limited diagnostic feedback. Evaluation and alignment are therefore shifting toward rubric-based assessment, which decomposes response quality into explicit, scoreable criteria. However, these criteria are not always reliable: they may demand information absent from the image, assert unverifiable motives, or require deliverables beyond what the task asks for. We term this phenomenon **Multimodal Rubric Hallucination (MRH)** and formalize rubric validity along two axes: Evidence Support and Task Fit. Unlike conventional multimodal hallucination in model responses, MRH makes the evaluation standard itself the object of diagnosis. We propose a new task, **MRH Diagnosis and Repair**: identify one of six MRH types and rewrite the hallucinated rubric into an evidence-supported, task-aligned replacement. We introduce **MRH-Bench**, comprising 8,472 verified multimodal instances organized into three progressively challenging tracks: rubric hallucination type identification, hallucinated rubric repair, and LLM–human evaluation agreement assessment. Across eight representative proprietary and open-weight MLLMs, the strongest prompted baseline reaches only 61.0% Macro-F1 and 15.80% repair success. We further propose **MRH-Guard**, a two-stage framework combining structured weighted supervised fine-tuning with **Quadrant-Aware Policy Optimization (QAPO)** to strengthen the diagnostic decisions that guide repair. MRH-Guard reaches 75.4% Macro-F1 (+14.4 points) and 19.20% repair success (+3.40 points).

Primary area:
datasets and benchmarks
Keywords:
Rubric-based Evaluation, Multimodal Large Language Models, Multimodal Hallucination, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_joJ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the validity of scoring rubrics in multimodal evaluation and introduces MRH-Bench to assess MLLMs’ ability to identify and repair erroneous rubrics. The authors further propose QAPO and train Qwen3-VL-8B through SFT followed by QAPO to obtain MRH-Guard. Experiments show that MRH-Guard outperforms existing baselines and approaches the performance of closed-source models.

Critical strengths
  1. The problem is practically meaningful. The paper investigates potential hallucinations in judging rubrics for multimodal evaluation, offering a useful perspective on improving the reliability of MLLM evaluation systems.
  2. The data construction and evaluation analyses are reasonably thorough. MRH-Bench is carefully designed, with generated samples verified by two independent models and three experts, and pre-adjudication agreement statistics reported. Appendix H additionally provides human calibration on 480 outputs and validation using a second judge on matched subsets. These experiments increase confidence in the automated evaluation results.
Critical weaknesses
  1. Potential mismatch between synthetic and real-world data. The benchmark constructs hallucinated rubrics by introducing specific errors into valid rubrics using an LLM. However, my core concern about this work is that the paper does not sufficiently characterize the manifestations, prevalence, or distribution of rubric hallucinations in real-world settings. Without analyzing naturally occurring rubric hallucinations and checking their consistency or similarity with the synthetic examples, the practical value of this work remains uncertain. The resulting benchmark may primarily measure performance on a narrowly defined, rule-based task for MLLM judges, with limited generalizability.
  2. The improvements from QAPO are limited, and their statistical reliability remains unclear. Although QAPO is well motivated, its gains over the GRPO baseline are modest. Moreover, the quantitative results for diagnostic classification and repair quality rely on LLM judges, raising concerns about reliability. Without reported training and inference variability or confidence intervals, it is difficult to establish whether the observed differences from GRPO are statistically reliable.
Questions for authors
  1. Can the authors characterize naturally occurring rubric hallucinations, including their prevalence and distribution across the six proposed categories? How closely do the synthetic errors in MRH-Bench resemble these real-world failures?
  2. Can the authors evaluate MRH-Guard on a separately collected set of naturally occurring erroneous rubrics to assess whether training on controlled synthetic injections generalizes beyond the benchmark’s construction procedure?
  3. Can the authors report results across multiple training seeds and paired confidence intervals for the main comparisons between QAPO and GRPO, particularly for classification Macro-F1, repair success, and downstream judge–human agreement?

If the authors convincingly address these concerns, I would consider raising my rating.

Reviewer_jR8gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies rubric-based multimodal evaluation and asks whether the evaluation criteria themselves are reliable. The authors term the failure Multimodal Rubric Hallucination (MRH): a rubric requirement that lacks Evidence Support (not verifiable from the image and task-authorized references) or Task Fit (beyond the prompt's task scope), formalized on two axes whose binary combinations yield four unit states (G/O/U/X) and six mutually exclusive MRH types (T1–T6). They propose a new task, MRH Diagnosis and Repair — identify the type and rewrite the affected scoring unit into an evidence-supported, task-aligned replacement while preserving grounded evaluative intent — and build MRH-Bench: 8,472 verified multimodal instances organized into three tracks (type identification; rubric repair; LLM–human agreement assessment under repaired rubrics). Construction proceeds in three stages: curating nine source datasets, injecting one controlled MRH per item (GPT-5.6Sol), repairing it blind (a second call seeing only the image, prompt, and the hallucinated rubric), collecting human scoring/ranking references under the gold repairs, and screening each candidate through two independent verifier models (Kimi-K3, Qwen3.8-Max) followed by a three-expert consensus with arbitration. They further propose MRH-Guard on a Qwen3-VL-8B LoRA backbone: a structured weighted SFT stage (diagnostic tags and type labels receive higher token weights than copied tokens) followed by Quadrant-Aware Policy Optimization (QAPO), which rewards per-axis accuracy, exact joint-quadrant accuracy, and group-relative Pareto dominance, each normalized per channel before combination. Across eight representative proprietary and open-weight MLLMs, the strongest prompted baseline leaves most of the headroom unfilled on all three tracks; MRH-Guard improves type identification and repair substantially, improves the downstream agreement measure modestly, and reports the sub-metrics on which GPT-5.5 remains stronger.

Critical strengths
  1. The problem choice is timely and important. Rubric-based scoring now underlies both model-based judging and rubric-driven reward training, and both assume the rubric is correct. Prior hallucination work checks model responses; this paper checks the criteria used to score them and, to the authors' knowledge, is the first benchmark of its kind. A wrong rubric does not fail quietly: a judge following it rewards hallucinated details and penalizes grounded ones (Figure 1). Six mechanisms with operational boundaries (Section 3.1; Appendix E) make the target checkable rather than rhetorical.
  2. Every layer of the work is verified, from benchmark labels to downstream use. On the data side, the one-mechanism-per-item design gives type identification a unique correct answer, and the blind-repair construction with verifier screening and expert adjudication yields quantified label and reference reliability (Section 3.2; Tables 8 and 10). On the evaluation side, an original-rubric control isolates the gain attributable to repair itself (Table 15), the repair auditor is calibrated against human judgments (Tables 13–14), the main conclusions replicate under a second judge on matched subsets (Table 16), and reward ablations with a second backbone test the training design rather than a single configuration (Tables 5, 12, and 17).
Critical weaknesses
  1. The reported incremental gains carry no uncertainty quantification.
  • All training runs use a single seed (seed 42, Appendix K), with no second run to bound training variance.
  • The margins the method's claim rests on are small: removing the Pareto reward changes Macro-F1 by 0.006, QAPO improves Macro-F1 by 0.013 and RS by 1.6pp over SFT+GRPO (Table 5), and the Spearman edge over GPT-5.5 is 0.0135 (Table 4).
  1. The benchmark tests controlled single-error injections, not rubric auditing in general.
  • Every evaluated hallucination is manufactured by a designated single-type injection from one generator (GPT-5.6 Sol, Section 3.2), so neither naturally occurring errors nor injections from an independent generator are tested.
  • The prompt states that each criterion contains exactly one MRH mechanism (Section 3.3; Appendix L), so the model is never asked whether a rubric is defective; false alarms and unwanted edits on valid criteria go unmeasured.
  • Because each item carries a single designated mechanism, multiple interacting errors are not measured.
Questions for authors
  1. How stable are the incremental QAPO gains? With a single training seed (Appendix K) and judge repetitions on fixed repairs (Appendix H.2), can you provide an uncertainty assessment for the method differences in Tables 4 and 5 (against GPT-5.5 and against SFT+GRPO, respectively), for example the dispersion across the three reported judge runs?
  2. How does the method behave when rubric validity is unknown? The prompts assume a known hallucination (Appendix L), and each benchmark item carries one designated injection (Section 3.2). Can you report performance on a held-out mixture containing valid criteria and errors outside the injection procedure, including false-positive edits of valid content?
  3. How sensitive is Track 2 to the canonical gold repair? The audit prompt declares the gold repair the canonical semantic specification (Appendix L), and human calibration uses that same gold (Appendix H.1). Can experts assess a sample of failed repairs against the task evidence without seeing the gold wording, and report how often an alternative repair is judged valid? This separates failure to repair from mismatch with one gold specification (Sem-Fid 21.67%, RS 19.20% in Table 2).
Reviewer_VnJaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a new benchmark, MRH-Bench, that measures Multimodal Rubric Hallucination (MRH). This problem is defined as errors in the rubic criteria that can lead to the LLM-as-a-judge to incorrectly give credit to inaccurate model responses. The paper create an evaluation protocol and tests existing models as well as a proposed MRH-Guard variant that was fine-tuned specifically for detecting rubic hallucinations.

Critical strengths

S1: The introduction of evaluation criteria hallucinations is novel and timely, especially in the context of automatically generated rubics as rewards such as RubicRM and V-Rubics, where manual audits of rubics is infeasible at scale.

Critical weaknesses

W1: The paper does not do a good job of justifying relevance of the problem.

  • Most of the focus lies on static rubic assessment which is a one-time curation task. The paper is most relevant when rubics are automatically generated, e.g. RubicRM and V-Rubics, but does identify this use-case specifically enough. That's likely also the only setting where the diagnose and repair regime makes sense, but it is not explicitly evaluated, e.g., does it improve downstream performance with rubic-reward trained models.
  • No evidence is provided for how often errors occur in rubics, whether they are human-written or AI generated. As a result, the paper does not establish whether this is an actively occurring problem.

W2: False-positive hallucination detection is considered.

  • Track 1 explicitly let's the model choose between hallucination types T1-T6, but never asks models to detect clean samples (it is not an output option at all). Importantly, models should be able to detect when samples are grounded so this should be an important part of the evaluation metric.
  • MRH-Guard is trained to always output a hallucination type and to always propose a correctly. If applied to a proper dataset, it would flag a lot of valid samples and try to rewrite all of them.

W3: The benchmark is potentially biased, but this is not tested.

  • GPT 5.6 Sol wrote the hallucination errors, and the gold repairs. A model that writes like GPT 5.6 Sol is directly rewarded through the comparison to the the gold repair set, especially in Track 2.
  • MRH-Guard is directly trained on samples generated with the same pipeline with GPT 5.6 Sol, so it directly benefits from this bias.
Questions for authors

I encourage the authors to address the aforementioned weaknesses. Specifically:

  • Can you provide some evidence that existing rubic evaluation datasets suffer from hallucinations?
  • How do the models perform when you evaluate on both clean and hallucinated samples together, and then report precision-recall on hallucination detection?
  • Can you generate a subset of the benchmark data with a different model family and then test whether the ranking of MRH-Guard holds up to control for any dataset bias?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.