acceptodds
Under review as a conference paper at ICLR 2027

SIGHT: Compositional Skill Routing for Instance-Specific Rubric Generation across Heterogeneous Multimodal Tasks

Abstract

Reward models guide reinforcement learning (RL) for multimodal large language models with scalar scores that obscure multi-dimensional human judgment criteria. To address this, rubric-as-reward decomposes response quality into structured, verifiable criteria. Despite rapid adoption, two questions remain: can benchmarks assess rubric fidelity to human-valued dimensions, and how can prompt-specific rubrics reflect human preferences across heterogeneous multimodal tasks? We introduce **MM-RubricBench**, the first multimodal rubric-guided reward benchmark, spanning six heterogeneous domains. Each curated question has a weighted expert-authored rubric and human ratings of eleven models' responses scored against it. These annotations support evaluation of judge-human preference alignment and generated rubric quality. We further propose **SIGHT**, compositional **S**kill routing for **I**nstance-specific rubric **G**eneration across **H**eterogeneous multimodal **T**asks. SIGHT transfers construction logic rather than rubric content: it extracts reusable skills from expert rubrics, builds instance-specific evaluation blueprints, selects complementary skills to cover them, and generates rubrics traceable to specific requirements. On MM-RubricBench, these rubrics outperform model-based scoring protocols in human-preference alignment, approaching the human rubric ceiling. When used as RL rewards, these rubrics also improve policies over baselines in multimodal reasoning, instruction following, and agentic rewarding. Construction-skill routing thus supports practical, scalable, and reliable multimodal alignment. Code, datasets, and benchmark will be publicly available.

Primary area:
reinforcement learning
Keywords:
Reward model, Rubric as reward, Multimodal large language model, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_e4JnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MM-RubricBench and SIGHT to address the evaluation and automatic generation of instance-specific rubrics for heterogeneous multimodal tasks. MM-RubricBench contains 305 prompts with expert-authored weighted rubrics and human ratings for 3,355 model responses. SIGHT is an LLM-based framework that extracts reusable rubric construction skills from expert rubrics and selects complementary skills to generate task-specific evaluation criteria. Experiments demonstrate improved agreement with human ratings and consistent downstream gains.

Critical strengths

1. Valuable benchmark contribution. MM-RubricBench is a carefully curated benchmark with rich annotations, including expert-authored weighted rubrics and multiple human-scored responses per prompt.

2. Strong presentation and empirical validation. The paper is well written, clearly organized, and easy to follow. The experiments are comprehensive, including comparisons with relevant baselines, ablation studies, and downstream RL evaluations.

Critical weaknesses

1. Limited methodological novelty. SIGHT is primarily an LLM-based pipeline combining established techniques, including prompting, clustering, retrieval, and greedy selection. This paper is more suitable for "Dataset & Benchmark" track rather than "reinforcement learning".

2. Indirect evaluation of rubric quality. MM-RubricBench primarily evaluates generated rubrics through the agreement between rubric-conditioned judge scores and human ratings, rather than directly assessing rubric quality. The evaluation does not fully establish whether generated rubrics faithfully capture human-valued criteria.

Questions for authors

MM-RubricBench contains only 305 problems, which appears relatively small compared with existing benchmarks. How do the authors assess the adequacy of this scale, particularly given the benchmark's coverage of six heterogeneous domains? Is this size sufficient to support reliable and representative evaluation?

3 comments

To Reviewer e4Jn

We thank Reviewer e4Jn for recognizing MM-RubricBench as a carefully curated benchmark with expert-authored weighted rubrics and multiple human-scored responses per prompt, and for finding the paper clearly organized with comprehensive experiments. We address each point below; all new results are marked in purple in the revised manuscript.

(W1-a) What SIGHT Transfers\color{#2980B9}{\textbf{(W1-a) What SIGHT Transfers}}

The key idea of SIGHT is to reuse how human experts build rubrics, rather than what their rubrics say. Existing rubric generators either write rubrics from scratch or retrieve human rubrics of related queries as in-context examples, as in RubricRAG. Retrieval transfers rubric content (the objects, values, and answer requirements of another question) which often do not apply to a new prompt.

SIGHT instead transfers the construction logic: how an expert splits requirements into independently checkable items, separates core from secondary requirements, assigns credits, and defines deductions. To make this logic reusable, we rewrite each human rubric as a construction recipe with instance-specific content removed, so that similarity reflects how a rubric is built rather than what it is about. For each new instance we then derive an evaluation blueprint of its requirements and errors, and select skills to cover that blueprint. The components are familiar, but the unit of reuse is not.

(W1-b) Why Reinforcement Learning Is the Primary Area\color{#2980B9}{\textbf{(W1-b) Why Reinforcement Learning Is the Primary Area}}

The paper focuses on reward construction for RL. Rubrics serve as reward functions: MM-RubricBench measures whether a reward agrees with human judgment, and SIGHT produces such rewards at scale. Used as RL rewards, SIGHT rubrics improve the base policy by 5.7, 4.3, and 8.0 points on multimodal reasoning, instruction following, and agentic rewarding, outperforming the other reward schemes in each setting. We place the work in reinforcement learning because reward design is central to that setting, and both the benchmark and the method are developed for this use.

 

(W2) Direct Human Evaluation of Rubric Quality\color{#2980B9}{\textbf{(W2) Direct Human Evaluation of Rubric Quality}}

Agreement between rubric-conditioned scores and human ratings is useful but indirect, so we also evaluate the rubrics directly. Three human annotators independently compare the rubrics of OpenRubrics, RubricRAG and SIGHT on all 305 prompts of MM-RubricBench, shown anonymously and in random order together with the image, question and reference answer (new Appendix C.5, Figure C.1).

Aspects. For each prompt they choose the best rubric on coverage (every core requirement has a scoring item, and no item asks for something the question does not), correctness (criteria and background knowledge agree with the image and the reference), and usability (credit is placed on the core requirements, equally correct answers can still earn it, and each item can be decided unambiguously). They also mark every rubric that misses or misinterprets a key requirement. Labels follow the majority vote, and prompts on which all three annotators disagree are settled by discussion.

Results. SIGHT is preferred most often on all three aspects: 116 prompts for coverage and 115 for correctness, against at most 102 and 98 for the two baselines. The margin is largest on usability, where SIGHT is chosen on 131 prompts (43.0%) against 92 and 82. Only 5 SIGHT rubrics (1.6%) miss or misinterpret a key requirement, compared with 12 (3.9%) for RubricRAG and 19 (6.2%) for OpenRubrics. We also review 600 rubrics sampled from the three RL training sets and find about 97% correct (new Appendix C.6).

(Q-a) What Makes a 305-Prompt Benchmark Reliable\color{#2980B9}{\textbf{(Q-a) What Makes a 305-Prompt Benchmark Reliable}}

Reliability here is better judged by how consistently a benchmark separates models than by prompt count alone. AIME, for example, has 30 problems per year and is still used to rank frontier models.

Existing multimodal reward benchmarks struggle to separate current frontier MLLMs: re-evaluating them on VL-RewardBench, MM-RLHF-RewardBench and Multimodal RewardBench yields inversions or ties in five of fifteen comparisons whose order is known in advance (Figure 2). MM-RubricBench resolves these cases: on 18 recent MLLMs, the human rubrics recover all 14 within-family release and flagship-versus-smaller-variant orderings, with an overall spread of 22.3 points (Table 2).

(Q-b) Resolution Across Model Scales and Prompt Budgets\color{#2980B9}{\textbf{(Q-b) Resolution Across Model Scales and Prompt Budgets}}

Scale resolution. On two open-weight families whose capability order is known in advance, InternVL3.5 (seven models, 1B to 241B) and Qwen3-VL (five models, 2B to 235B), scores rise at every size step (Kendall τ\tau = 1.0), with spreads of 35.1 and 45.6 points (new Appendix A.7, Table A.8).

Signal per prompt. Every response is scored on a weighted 0-5 rubric rather than as correct or incorrect. On random subsets of Qwen3-VL prompts, graded scores separate 67.3% of model pairs significantly with only 25 prompts and 91.4% with 100, against 45.1% and 84.0% for the same scores reduced to binary labels (Table A.9).

Stability. Two disjoint halves of 152 prompts agree with Kendall τb\tau_b = 0.79 (Appendix A.6), and a second judge, Gemini 3.1 Pro, reproduces the ranking with τb\tau_b = 0.90 and Spearman ρ\rho = 0.97.

(Q-c) Six Heterogeneous Domains: Resolution and Robustness\color{#2980B9}{\textbf{(Q-c) Six Heterogeneous Domains: Resolution and Robustness}}

Because the benchmark spans heterogeneous domains, we also report what the data resolve inside each of them (new Appendix A.8).

Domain-level uncertainty. For three model pairs we report score differences per domain with pointwise 95% confidence intervals from 20,000 paired bootstrap resamples of prompts. Seed2.0 Pro scores 5.84 points above Seed1.8 overall (95% CI [1.77, 9.97] on the 0-100 scale), with a larger difference of 11.73 points in Structured Information (95% CI [2.67, 20.80]). These intervals show where the overall ranking is clearest and where domain-level differences remain narrower.

Weighting and composition. Using the raw judge scores for all 18 models, prompt-weighted and equal-domain averaging agree with Kendall τb\tau_b = 0.9705 and Spearman ρ\rho = 0.9954, with no model moving by more than one rank. Removing each domain in turn gives τb\tau_b = 0.9312 to 0.9836 and preserves 96.08% to 98.69% of pair orders, and the top-three set never changes. The aggregate ranking remains stable under these domain-composition checks.

(Q-d) Task Coverage and the Scope of Our Claim\color{#2980B9}{\textbf{(Q-d) Task Coverage and the Scope of Our Claim}}

MM-RubricBench is a curated cross-domain evaluation suite spanning six domains and 3535 subcategories: structured information analysis, comprehension, visual reasoning, interaction, recognition, and knowledge-intensive tasks (Appendix A.1), over visual inputs such as charts, documents, posters, screenshots, and natural photographs.

Each prompt requires visual grounding and combines multiple constraints, for example pairing recognition with reasoning or extraction with formatting rules (Appendix A.2). The domain mix is chosen for broad task coverage rather than to mirror real-world task frequencies. Together with the analyses above, this supports using the 305 prompts for aggregate cross-domain evaluation.

 

We thank Reviewer e4Jn again for the comments on the unit of reuse behind SIGHT and on the adequacy of the benchmark scale. They motivated the direct human evaluation of rubric quality and the new domain-level resolution analyses. We hope the added analyses speak to your remaining questions, and we would be grateful if you would reconsider your score.

AuthorsOpenReview ↗

Follow-up to Reviewer e4Jn

Dear Reviewer e4Jn,

Thanks again for the constructive feedback. With the discussion period nearly over, we would be grateful if you could check our response when convenient.

We clarified that SIGHT transfers construction logic rather than rubric content, and that both the benchmark and the method serve reward construction for RL, where SIGHT improves the base policy by 5.7, 4.3, and 8.0 points across three settings. On rubric quality, three annotators directly compare anonymised rubrics on all 305 prompts: SIGHT is preferred most often on coverage, correctness, and usability, with only 1.6% missing or misinterpreting a key requirement. On scale, scores rise at every size step on two open-weight families, graded scores separate models with far fewer prompts than binary labels, and domain-level bootstrap and leave-one-domain-out checks show that the aggregate ranking is stable across domains.

We hope the added analyses speak to your remaining questions. If so, we would be grateful for your continued support. We would be glad to discuss further. Thank you again for your time.

Reviewer_A5PVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper addresses the opacity of scalar rewards in multimodal alignment through rubric-based evaluation. The authors introduce MM-RubricBench, a multi-domain benchmark with expert-crafted rubrics and human annotations to evaluate preference alignment. Additionally, they propose SIGHT, a skill-routing framework that generates instance-specific rubrics. Experiments demonstrate superior human-preference alignment and notable RL gains across diverse multimodal reasoning tasks.

Critical strengths
  1. The paper innovatively decouples the rubric construction logic from instance-specific factual content. By leveraging skill card extraction and blueprint residual routing mechanisms, it establishes a clear and interpretable paradigm for automatically generating structured rubrics across diverse domains.

  2. Moving beyond conventional binary assessments, the work introduces a fine-grained reward design incorporating graded scores and penalty caps. It achieves consistent and substantial policy improvements across three heterogeneous RL tasks, including multimodal reasoning, multi-turn instruction following, and agent tool use, effectively demonstrating the practical value of rubrics as dense reward signals.

Critical weaknesses
  1. SIGHT explicitly requires standard reference answers across evaluation blueprint generation, rubric construction, and final RL judge scoring. This makes it inapplicable to real-world open-domain RLHF alignment scenarios where ground truth is unavailable, substantially limiting its practical value.

  2. The benchmark contains only 305 prompts across 6 domains and 35 subcategories, which translates to fewer than 9 prompts per subcategory on average. Such a small sample size lacks sufficient statistical confidence, making it difficult to justify the benchmark as an authoritative evaluation standard for frontier MLLMs.

  3. The framework incurs heavy computational costs due to multiple consecutive LLM calls for blueprint extraction, skill routing, rubric generation, and multi-step verification and repair. In addition, because the downstream 72B RL judge takes both reference answers and background knowledge as input, it remains unclear whether the observed gains come from the rubric logic itself or simply from the supplementary factual context.

Questions for authors

see Critical Weaknesses.

6 comments

To Reviewer A5PV

We thank Reviewer A5PV for recognizing that SIGHT decouples the rubric construction logic from instance-specific factual content through skill cards and blueprint residual routing, and that the graded rewards with penalty caps deliver consistent policy improvements across three heterogeneous RL tasks. We address each weakness below; all new results are marked in purple in the revised manuscript.

(W1-a) No Unique Ground Truth Is Needed\color{#2980B9}{\textbf{(W1-a) No Unique Ground Truth Is Needed}}

A reference enters SIGHT at two stages, rubric construction and reward scoring, and at both stages a high-quality response is enough. For open-ended tasks such as instruction following, the IF experiment uses the preferred response of the MM-IFEngine preference data as one good answer among many, and the agentic setting supplies only the index of the preferred candidate. SIGHT improves the base policy by 4.3 and 8.0 points in these two settings (Tables C.1 and C.2). On the open-ended VQA benchmarks MM-Vet v2 and KOFFVQA, a correctness-only judge given the same reference changes the base policy by −0.6-0.6 and +0.4+0.4, while SIGHT gains 2.2 and 3.2. For RL data with only image-text inputs, a good model response can serve the same role.

(W1-b) Scoring Without the Reference Answer\color{#2980B9}{\textbf{(W1-b) Scoring Without the Reference Answer}}

At reward time, the evidence part of a SIGHT rubric already records the facts needed for evaluation, and the judge checks factual claims against this background before consulting the reference (Appendix D.6). Retraining with the reference removed from the judge prompt and all other settings unchanged yields 67.3 (+5.4) on multimodal reasoning, close to 67.5 (+5.7) with the reference and well above the correctness judge at 63.5 (+1.7) (new Appendix C.4, Table C.4). The reference remains useful, but most of the reward signal is already carried by the rubric.

(W1-c) Prompts with Multiple Valid Answers\color{#2980B9}{\textbf{(W1-c) Prompts with Multiple Valid Answers}}

When a question admits several valid answers, SIGHT keeps the available reference from becoming a mandatory template. The instantiation step is asked to state acceptable equivalent answers wherever a requirement admits them (Appendix B.4), and the skill bank inherits this practice from expert rubrics: S12 accepts equivalent solution paths, and S11 gives partial credit to near-miss expressions (Appendix Table B.1). In MM-RubricBench, 77 of the 305 prompts are creation tasks without a single correct answer. In one such case the reference is a full story about a rabbit sculpture; the SIGHT rubric credits identifying the rabbit, keeping it as the theme, and the 300-character limit, without turning the story content into requirements. Across the benchmark, 40% of SIGHT rubrics explicitly list accepted alternatives or equivalent expressions. We add two such cases to Appendix E.3 (Figure E.7).

(W2-a) What Makes a Reward Benchmark Reliable\color{#2980B9}{\textbf{(W2-a) What Makes a Reward Benchmark Reliable}}

Reliability here is better judged by how consistently a benchmark separates models than by prompt count alone. AIME, for example, has 30 problems per year and is still used to rank frontier models.

Existing multimodal reward benchmarks struggle to separate current frontier MLLMs: re-evaluating them on VL-RewardBench, MM-RLHF-RewardBench and Multimodal RewardBench yields inversions or ties in five of fifteen comparisons whose order is known in advance, such as a release against its successor (Figure 2). MM-RubricBench resolves these cases: on 18 recent MLLMs, the human rubrics recover all 14 within-family release and flagship-versus-smaller-variant orderings, including the pairs that the existing benchmarks invert or tie, with an overall spread of 22.3 points (Table 2).

(W2-b) Resolution Across Model Scales and Prompt Budgets\color{#2980B9}{\textbf{(W2-b) Resolution Across Model Scales and Prompt Budgets}}

Scale resolution. We score two open-weight families whose capability order is known in advance: InternVL3.5 (seven models, 1B to 241B) and Qwen3-VL (five models, 2B to 235B). Scores rise at every size step in both families (Kendall τ\tau = 1.0), with spreads of 35.1 and 45.6 points (new Appendix A.7, Table A.8).

Signal per prompt. Every response is scored on a weighted 0-5 rubric rather than as correct or incorrect. On random subsets of Qwen3-VL prompts, graded scores separate 67.3% of model pairs significantly with only 25 prompts and 91.4% with 100 prompts, against 45.1% and 84.0% when the same scores are reduced to correct or incorrect. Graded scores separate all pairs from 250 prompts on, whereas the binary scores need the full set of 305 (Table A.9).

Stability. Two disjoint halves of 152 prompts already agree with Kendall τb\tau_b = 0.79 (Appendix A.6), and rescoring the 18 models with Gemini 3.1 Pro instead of Doubao Seed2.0 gives τb\tau_b = 0.90 and Spearman ρ\rho = 0.97, with the top seven models in the same order.

(W2-c) Domain-Level Resolution and Robustness to Domain Composition\color{#2980B9}{\textbf{(W2-c) Domain-Level Resolution and Robustness to Domain Composition}}

We also report what the data resolve inside each domain of MM-RubricBench (new Appendix A.8).

Domain-level uncertainty. For three model pairs we report score differences separately in each domain, with pointwise 95% confidence intervals from 20,000 paired bootstrap resamples of prompts, where both models are always evaluated on the same sampled prompts. Seed2.0 Pro scores 5.84 points above Seed1.8 overall (95% CI [1.77, 9.97] on the 0-100 scale), with a larger difference of 11.73 points in Structured Information (95% CI [2.67, 20.80]). These intervals show where the overall ranking is clearest and where domain-level differences remain narrower.

Weighting and composition. Using the raw judge scores for all 18 models, averaging all prompts and averaging the six domain means equally give Kendall τb\tau_b = 0.9705 and Spearman ρ\rho = 0.9954; 150 of 153 model pairs keep the same order or tie status, and no model moves by more than one rank. Removing each domain in turn gives τb\tau_b = 0.9312 to 0.9836 and preserves 96.08% to 98.69% of pair orders, with a maximum rank change of two, and the top-three set is unchanged throughout. The aggregate ranking remains stable under these domain-composition checks.

(W2-d) Task Coverage and the Scope of Our Claim\color{#2980B9}{\textbf{(W2-d) Task Coverage and the Scope of Our Claim}}

MM-RubricBench is a curated cross-domain evaluation suite spanning six domains and 3535 subcategories: structured information analysis, comprehension, visual reasoning, interaction, recognition, and knowledge-intensive tasks (Appendix A.1), over visual inputs such as charts, documents, posters, screenshots, and natural photographs.

Each prompt requires visual grounding and combines multiple constraints, for example pairing recognition with reasoning or extraction with formatting rules (Appendix A.2). The domain mix is chosen for broad task coverage rather than to mirror real-world task frequencies. Together with the analyses above, this supports using the 305 prompts for aggregate cross-domain evaluation.

(W3-a) Computational Cost of Rubric Construction and RL\color{#2980B9}{\textbf{(W3-a) Computational Cost of Rubric Construction and RL}}

All LLM calls for building rubrics happen offline before training. The skill bank is built once from the expert rubrics, and for each training instance the rubric is constructed once, stored, and reused for every rollout and epoch (Appendix D.2). Construction generally takes three LLM calls per instance, for blueprint extraction, routing, and generation. Verification uses rule and embedding checks without LLM calls, and the single repair call is triggered for only 8.2% of the MM-RubricBench prompts. Instances are independent, so construction parallelises fully across the dataset.

During RL the reward costs one judge call per sampled response, the same as any LLM-as-a-judge reward in the baseline settings, including the correctness-judge baseline. With 8 or 16 rollouts per prompt, the offline cost of about three calls per prompt is amortised over every rollout and epoch of training.

(W3-b) The Reference Alone Does Not Explain the Gains\color{#2980B9}{\textbf{(W3-b) The Reference Alone Does Not Explain the Gains}}

The correctness-only LLM-judge baseline scores each response against the same reference answer without a rubric. It improves the base policy by 1.7 points on multimodal reasoning and 2.1 points on instruction following, while SIGHT improves it by 5.7 and 4.3 points (Table 4 and Table C.1).

(W3-c) The Factual Context Alone Does Not Explain the Agreement\color{#2980B9}{\textbf{(W3-c) The Factual Context Alone Does Not Explain the Agreement}}

In the component ablations of Appendix C.4 the rubric is still generated from the same evaluation blueprint, so its background knowledge is built from the same evidence and only the construction logic is removed. Without skill guidance, pairwise accuracy drops from 0.73 to 0.62, and without the construction procedures it drops to 0.65.

(W3-d) A Fact-Only Reward in RL\color{#2980B9}{\textbf{(W3-d) A Fact-Only Reward in RL}}

To separate the contribution of facts from that of criteria in RL, we build a reward in which the judge receives the reference answer and the background knowledge of the SIGHT rubric, but none of its criteria or deductions, and assigns a 0-5 score. All other settings follow the multimodal reasoning experiment (new Appendix C.4, Table C.4).

RewardAvg.Δ\Delta
Qwen3-VL-4B-Instruct61.90
RL w/ correctness LLM-judge63.5+1.7
RL w/ fact-only reward65.5+3.7
RL w/o reference in judge prompt67.3+5.4
RL w/ SIGHT rubrics67.5+5.7

The fact-only reward sits above the correctness judge because the blueprint stage records evaluation facts as evidence and carries them into the rubric as background knowledge, so the judge receives these facts in structured form rather than inferring them from the reference alone. The criteria and deductions still matter: they state what earns credit, how much each requirement is worth, and which errors are penalised. Without them the reward recovers 3.7 of the 5.7 points, so structured facts and construction logic contribute jointly.

 

We thank Reviewer A5PV again for the comments on the reference requirement, the statistical resolution of the benchmark, and the attribution of the RL gains. These suggestions motivated the reference-ablated and fact-only reward analyses and the new domain-level checks. We hope the additional results address your concerns, and we would be grateful if you would reconsider your score.

Reviewer_gRgvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces MM-RubricBench, a multimodal rubric benchmark containing 305 questions, expert-written rubrics, and human ratings of responses from 11 models for each question. It also proposes SIGHT, which extracts reusable rules from expert rubrics, then selects and combines them based on a new question’s image, text, and reference answer to generate a question-specific rubric. The authors evaluate the generated rubrics through agreement with human ratings, ablation studies, and three types of downstream reinforcement learning tasks.

Critical strengths
  1. MM-RubricBench provides both expert-written rubrics and human ratings of responses. Compared with datasets containing only preference labels, it makes it easier to examine whether models assess the right aspects of response quality, providing a useful resource for research on multimodal rubrics.

  2. SIGHT’s attempt to reuse experts’ approaches to rubric construction has some merit. Compared with RubricRAG, its generated rubrics improve pairwise agreement with human rankings from 70% to 73%. Improvements across three types of downstream reinforcement learning tasks also support its practical utility.

Critical weaknesses
  1. The human annotation procedure is not sufficiently described. The paper outlines the scoring process, but should clarify how many annotators rated each response, how disagreements were resolved, and the level of inter-annotator agreement, since human ratings serve as a key reference in the main experiments.

  2. The evaluation of rubric quality is limited. A good rubric should be clear, actionable, and capture the main requirements of the question. The paper evaluates scoring agreement only with Seed2.0 as the judge; it remains unclear whether the gains hold with other judges. The rubric generator’s understanding of the image and text also directly affects rubric quality, but the paper does not examine whether it misses or misinterprets key requirements, or compare different generator models.

  3. Requiring a reference answer limits the method’s applicability. For open-ended questions, reference answers can be difficult to construct, and multiple answers may be valid. The authors should clarify whether the generated rubrics turn features of a particular reference answer into mandatory requirements, potentially overlooking or penalizing other equally good or better responses.

Questions for authors
  1. Could the authors provide the number of annotators per response, how disagreements were resolved, and the results of inter-annotator agreement and independent preference checks?

  2. Have the authors tried different judges or rubric-generation models? If so, how did the results change? If a manual review was conducted, it would also be helpful to know whether the generated rubrics missed or misinterpreted any key requirements.

  3. Could the authors clarify how the reference answers were obtained and checked? For questions with multiple valid answers, how does the method avoid making the rubric overly dependent on one reference answer? Would it be difficult to apply the method to RL datasets that provide only image–text inputs without reference answers?

5 comments

To Reviewer gRgv

We thank Reviewer gRgv for the careful reading and for recognizing that MM-RubricBench supplies both expert-written rubrics and human ratings of model responses, and that reusing experts' approaches to rubric construction has merit. We address each weakness and question below; all new results are marked in purple in the revised manuscript.

(W1-a) Annotation Team and Per-Response Scoring\color{#2980B9}{\textbf{(W1-a) Annotation Team and Per-Response Scoring}}

Annotation is carried out by a team of 18 annotators with extensive data-annotation experience; all of them complete training and a trial annotation round before the formal annotation. Each response is scored by one annotator against the rubric of its prompt, so the scoring basis stays the same across the eleven responses of that prompt. These details are now in Appendix A.3.

(W1-b) Verification and Disagreement Resolution\color{#2980B9}{\textbf{(W1-b) Verification and Disagreement Resolution}}

Every score is checked by a verifier other than the annotator who assigned it. A score that fails verification is returned to the original annotator for revision and checked again by the verifier. Disagreements are resolved by revision under review.

(W1-c) Independent Preference Check and Its Agreement\color{#2980B9}{\textbf{(W1-c) Independent Preference Check and Its Agreement}}

During benchmark construction, we randomly sample five response pairs for every prompt, 1,525 pairs in total. An annotator other than the one who scored the responses judges which response in each pair is better, without seeing the scores. At the first check, this judgment agrees with the score-induced preference on 94% of the pairs. The remaining pairs are discussed with the original annotator; scores that need correction are revised by that annotator, and any needed rubric revision is made by the original expert, then verified again. This preference-level check supports the reliability of the human scores used as the reference in our main experiments.

(W2-a) A Second Judge for Capability Ranking and Protocol Comparison\color{#2980B9}{\textbf{(W2-a) A Second Judge for Capability Ranking and Protocol Comparison}}

We rescore the 18 models of Table 2 with Gemini 3.1 Pro in place of Doubao Seed2.0, keeping the human rubrics and the scoring protocol unchanged (new Appendix A.7, Table A.6). The two judges give Kendall τb\tau_b = 0.90 and Spearman ρ\rho = 0.97 and order 94.8% of the model pairs identically. The top seven models keep the same order, and all 14 release-order checks still hold.

We also rerun the comparison of Table 3 with Gemini 3.1 Pro as the judge, keeping the rubrics of every protocol unchanged (new Table A.7). The protocols keep the order of Table 3 on all five metrics, and SIGHT again outperforms every model-based protocol on each of them. Against RubricRAG, SIGHT lowers MAE from 0.97 to 0.86 and raises pairwise accuracy from 0.69 to 0.75.

(W2-b) Changing the Rubric Construction Model\color{#2980B9}{\textbf{(W2-b) Changing the Rubric Construction Model}}

The training rubrics are built with Gemini-2.5-Pro by default (Appendix D.2). To check sensitivity to this choice, we rebuild them with the open-weight Qwen3.5-122B-A10B and rerun RL on multimodal reasoning and instruction following, leaving all other settings unchanged (new Appendix C.4, Table C.5).

Construction modelReasoning Avg.IF Avg.
Gemini-2.5-Pro (default)67.5 (+5.7)76.4 (+4.3)
Qwen3.5-122B-A10B67.4 (+5.6)76.1 (+4.0)

Both runs remain above the correctness LLM-judge and RubricRAG rewards of Table 4 and Table C.1, suggesting that the transferred construction logic, rather than a particular generator, accounts for most of the gains.

(W2-c) Human Review of the Generated Rubrics\color{#2980B9}{\textbf{(W2-c) Human Review of the Generated Rubrics}}

Beyond the agreement between rubric-conditioned scores and human ratings, we also assess the rubrics directly. Three annotators independently compare the rubrics of OpenRubrics, RubricRAG and SIGHT on all 305 prompts, shown anonymously and in random order together with the image, question and reference answer (new Appendix C.5, Figure C.1).

Aspects. For each prompt they choose the best rubric on coverage (every core requirement has a scoring item, and no item asks for something the question does not), correctness (criteria and background knowledge agree with the image and the reference), and usability (credit is placed on the core requirements, equally correct answers can still earn it, and each item can be decided unambiguously). They also mark every rubric that misses or misinterprets a key requirement. Labels follow the majority vote, and prompts on which all three annotators disagree are settled by discussion.

Results. SIGHT is preferred most often on all three aspects: 116 prompts for coverage and 115 for correctness, against at most 102 and 98 for the two baselines. The margin is largest on usability, where SIGHT is chosen on 131 prompts (43.0%), against 92 for RubricRAG and 82 for OpenRubrics. Only 5 SIGHT rubrics (1.6%) miss or misinterpret a key requirement, compared with 12 (3.9%) for RubricRAG and 19 (6.2%) for OpenRubrics.

(W2-d) Human Review of the Training Rubrics\color{#2980B9}{\textbf{(W2-d) Human Review of the Training Rubrics}}

We also review the rubrics that SIGHT constructs for the three RL configurations, randomly sampling 200 rubrics from each of the reasoning, IF and agentic training sets (new Appendix C.6). About 97% are judged correct. All of them are well formed, every one covers the core requirements of its question, and no misreading of the question is found. The main remaining issue is in the agentic data, where roughly 0.7% of rubrics favor the candidate opposite to both the label and the reference; this appears to stem partly from ambiguous prompts or subjective reference preferences, and filtering such instances may further improve training data quality.

(W3-a) No Unique Ground Truth Is Needed\color{#2980B9}{\textbf{(W3-a) No Unique Ground Truth Is Needed}}

A reference enters SIGHT at two stages, rubric construction and reward scoring. For open-ended tasks such as instruction following, a high-quality response is enough: the IF experiment uses the preferred response of the MM-IFEngine preference data as one good answer among many, and the agentic setting supplies only the index of the preferred candidate. SIGHT improves the base policy by 4.3 and 8.0 points in these two settings (Tables C.1 and C.2). On the open-ended VQA benchmarks MM-Vet v2 and KOFFVQA, a correctness-only judge given the same reference changes the base policy by −0.6-0.6 and +0.4+0.4, while SIGHT gains 2.2 and 3.2. For RL data with only image-text inputs, a good model response can serve the same role.

(W3-b) Scoring Without the Reference Answer\color{#2980B9}{\textbf{(W3-b) Scoring Without the Reference Answer}}

At reward time, the evidence part of a SIGHT rubric already records the facts needed for evaluation, and the judge checks factual claims against this background before consulting the reference (Appendix D.6). Retraining with the reference removed from the judge prompt and all other settings unchanged yields 67.3 (+5.4) on multimodal reasoning, close to 67.5 (+5.7) with the reference and well above the correctness judge at 63.5 (+1.7) (new Appendix C.4, Table C.4).

(W3-c) The Reference Is Evidence, Not a Template\color{#2980B9}{\textbf{(W3-c) The Reference Is Evidence, Not a Template}}

During blueprint extraction, facts in the reference are recorded as evidence and passed to the rubric as background knowledge, and only what the question directly asks for becomes a credited criterion (Appendix B.2: "additional facts in a reference answer may remain evidence without becoming credited requirements"). Of the 1,046 positive requirements of SIGHT rubrics on MM-RubricBench, 63% are grounded in the image, 15% in the instruction, and 22% in the reference.

(W3-d) Prompts with Multiple Valid Answers\color{#2980B9}{\textbf{(W3-d) Prompts with Multiple Valid Answers}}

The instantiation step is asked to state acceptable equivalent answers wherever a requirement admits them (Appendix B.4), and the skill bank inherits this practice from expert rubrics: S12 accepts equivalent solution paths, and S11 gives partial credit to near-miss expressions (Appendix Table B.1). In MM-RubricBench, 77 of the 305 prompts are creation tasks without a single correct answer. In one such case the reference is a full story about a rabbit sculpture; the SIGHT rubric credits identifying the rabbit, keeping it as the theme, and the 300-character limit, without turning the story content into requirements. Across the benchmark, 40% of SIGHT rubrics explicitly list accepted alternatives or equivalent expressions. We add two such cases to Appendix E.3 (Figure E.7).

 

(Q1) Annotators per Response, Disagreement Resolution and Agreement\color{#2980B9}{\textbf{(Q1) Annotators per Response, Disagreement Resolution and Agreement}}

One annotator scores each response, a different verifier checks every score, and failed checks are revised by the original annotator (W1-a, W1-b). The independent preference check on 1,525 pairs agrees with the score-induced preference on 94% at the first check (W1-c). These details are now in Appendix A.3.

(Q2) Different Judges, Different Generators, and the Manual Review\color{#2980B9}{\textbf{(Q2) Different Judges, Different Generators, and the Manual Review}}

We added all three experiments. A second judge preserves both the capability ranking (τb\tau_b = 0.90) and the protocol ordering (W2-a); an open-weight construction model reproduces the RL gains (W2-b); and the manual review finds missed or misinterpreted requirements in 1.6% of SIGHT rubrics against 3.9% and 6.2% for the baselines, with about 97% of sampled training rubrics judged correct (W2-c, W2-d).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive comments.

We are glad that the reviewers find MM-RubricBench a valuable resource with expert-written rubrics and human ratings (gRgv, e4Jn), find the separation of construction logic from instance-specific content clear and interpretable (A5PV), and note the consistent gains of SIGHT rubrics as RL rewards across reasoning, instruction following, and agentic rewarding (gRgv, A5PV, e4Jn).

We address each point below and revise the manuscript accordingly, with all changes marked in purple. The main additions are as follows.

  1. Robustness and resolution of the benchmark (new Appendices A.7 and A.8). Rescoring the 18 models of Table 2 with Gemini 3.1 Pro as a second judge yields Kendall τ_b = 0.90 with the original ranking, and the same judge preserves the protocol order of Table 3, with SIGHT best on every metric (Tables A.6 and A.7). On two open-weight families with a known capability order, scores rise at every size step, and graded rubric scores separate models with far fewer prompts than binary labels (Tables A.8 and A.9). Domain-level paired bootstrap intervals and leave-one-domain-out checks further show that the aggregate ranking is stable across domains and under equal-domain averaging (Tables A.10 to A.12).
  2. Direct human evaluation of generated rubrics (new Appendix C.5). On all 305 prompts, three annotators compare the anonymised rubrics of OpenRubrics, RubricRAG and SIGHT. SIGHT is preferred most often on coverage, correctness and usability; only 1.6% of its rubrics miss or misinterpret a key requirement, compared with 3.9% and 6.2% for the two baselines (Figure C.1).
  3. Review of training rubrics (new Appendix C.6). Of 600 rubrics sampled from the three RL training sets, about 97% are judged correct; we also report the remaining issues.
  4. Annotation details (Appendix A.3). We describe the annotation team, single scoring with independent verification, the writing and checking of reference answers, and an independent preference check on 1,525 response pairs that agrees with the score-induced preference on 94% of them.
  5. Open-ended prompts (Appendix E.3). New cases show that SIGHT treats the reference as one valid response and keeps its content as background rather than as mandatory requirements (Figure E.7).
  6. Attribution of the RL gains (new Table C.4 in Appendix C.4). A fact-only reward that keeps the background knowledge of the SIGHT rubric but removes its criteria and deductions recovers 3.7 of the 5.7 points on multimodal reasoning, against 1.7 for the correctness judge. Removing the reference answer from the judge prompt leaves the gain nearly unchanged at +5.4, indicating that most of the signal comes from the rubric itself.
  7. Construction model (new Table C.5 in Appendix C.4). Rebuilding the training rubrics with the open-weight Qwen3.5-A10B yields gains close to the default setting: +5.6 and +4.0 on multimodal reasoning and instruction following, against +5.7 and +4.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.