- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MM-RubricBench and SIGHT to address the evaluation and automatic generation of instance-specific rubrics for heterogeneous multimodal tasks. MM-RubricBench contains 305 prompts with expert-authored weighted rubrics and human ratings for 3,355 model responses. SIGHT is an LLM-based framework that extracts reusable rubric construction skills from expert rubrics and selects complementary skills to generate task-specific evaluation criteria. Experiments demonstrate improved agreement with human ratings and consistent downstream gains.
Critical strengths
1. Valuable benchmark contribution. MM-RubricBench is a carefully curated benchmark with rich annotations, including expert-authored weighted rubrics and multiple human-scored responses per prompt.
2. Strong presentation and empirical validation. The paper is well written, clearly organized, and easy to follow. The experiments are comprehensive, including comparisons with relevant baselines, ablation studies, and downstream RL evaluations.
Critical weaknesses
1. Limited methodological novelty. SIGHT is primarily an LLM-based pipeline combining established techniques, including prompting, clustering, retrieval, and greedy selection. This paper is more suitable for "Dataset & Benchmark" track rather than "reinforcement learning".
2. Indirect evaluation of rubric quality. MM-RubricBench primarily evaluates generated rubrics through the agreement between rubric-conditioned judge scores and human ratings, rather than directly assessing rubric quality. The evaluation does not fully establish whether generated rubrics faithfully capture human-valued criteria.
Questions for authors
MM-RubricBench contains only 305 problems, which appears relatively small compared with existing benchmarks. How do the authors assess the adequacy of this scale, particularly given the benchmark's coverage of six heterogeneous domains? Is this size sufficient to support reliable and representative evaluation?
3 comments
Reliability here is better judged by how consistently a benchmark separates models than by prompt count alone. AIME, for example, has 30 problems per year and is still used to rank frontier models.
Existing multimodal reward benchmarks struggle to separate current frontier MLLMs: re-evaluating them on VL-RewardBench, MM-RLHF-RewardBench and Multimodal RewardBench yields inversions or ties in five of fifteen comparisons whose order is known in advance (Figure 2). MM-RubricBench resolves these cases: on 18 recent MLLMs, the human rubrics recover all 14 within-family release and flagship-versus-smaller-variant orderings, with an overall spread of 22.3 points (Table 2).
Scale resolution. On two open-weight families whose capability order is known in advance, InternVL3.5 (seven models, 1B to 241B) and Qwen3-VL (five models, 2B to 235B), scores rise at every size step (Kendall = 1.0), with spreads of 35.1 and 45.6 points (new Appendix A.7, Table A.8).
Signal per prompt. Every response is scored on a weighted 0-5 rubric rather than as correct or incorrect. On random subsets of Qwen3-VL prompts, graded scores separate 67.3% of model pairs significantly with only 25 prompts and 91.4% with 100, against 45.1% and 84.0% for the same scores reduced to binary labels (Table A.9).
Stability. Two disjoint halves of 152 prompts agree with Kendall = 0.79 (Appendix A.6), and a second judge, Gemini 3.1 Pro, reproduces the ranking with = 0.90 and Spearman = 0.97.
Because the benchmark spans heterogeneous domains, we also report what the data resolve inside each of them (new Appendix A.8).
Domain-level uncertainty. For three model pairs we report score differences per domain with pointwise 95% confidence intervals from 20,000 paired bootstrap resamples of prompts. Seed2.0 Pro scores 5.84 points above Seed1.8 overall (95% CI [1.77, 9.97] on the 0-100 scale), with a larger difference of 11.73 points in Structured Information (95% CI [2.67, 20.80]). These intervals show where the overall ranking is clearest and where domain-level differences remain narrower.
Weighting and composition. Using the raw judge scores for all 18 models, prompt-weighted and equal-domain averaging agree with Kendall = 0.9705 and Spearman = 0.9954, with no model moving by more than one rank. Removing each domain in turn gives = 0.9312 to 0.9836 and preserves 96.08% to 98.69% of pair orders, and the top-three set never changes. The aggregate ranking remains stable under these domain-composition checks.
MM-RubricBench is a curated cross-domain evaluation suite spanning six domains and subcategories: structured information analysis, comprehension, visual reasoning, interaction, recognition, and knowledge-intensive tasks (Appendix A.1), over visual inputs such as charts, documents, posters, screenshots, and natural photographs.
Each prompt requires visual grounding and combines multiple constraints, for example pairing recognition with reasoning or extraction with formatting rules (Appendix A.2). The domain mix is chosen for broad task coverage rather than to mirror real-world task frequencies. Together with the analyses above, this supports using the 305 prompts for aggregate cross-domain evaluation.
We thank Reviewer e4Jn again for the comments on the unit of reuse behind SIGHT and on the adequacy of the benchmark scale. They motivated the direct human evaluation of rubric quality and the new domain-level resolution analyses. We hope the added analyses speak to your remaining questions, and we would be grateful if you would reconsider your score.
Follow-up to Reviewer e4Jn
Dear Reviewer e4Jn,
Thanks again for the constructive feedback. With the discussion period nearly over, we would be grateful if you could check our response when convenient.
We clarified that SIGHT transfers construction logic rather than rubric content, and that both the benchmark and the method serve reward construction for RL, where SIGHT improves the base policy by 5.7, 4.3, and 8.0 points across three settings. On rubric quality, three annotators directly compare anonymised rubrics on all 305 prompts: SIGHT is preferred most often on coverage, correctness, and usability, with only 1.6% missing or misinterpreting a key requirement. On scale, scores rise at every size step on two open-weight families, graded scores separate models with far fewer prompts than binary labels, and domain-level bootstrap and leave-one-domain-out checks show that the aggregate ranking is stable across domains.
We hope the added analyses speak to your remaining questions. If so, we would be grateful for your continued support. We would be glad to discuss further. Thank you again for your time.
To Reviewer e4Jn
We thank Reviewer e4Jn for recognizing MM-RubricBench as a carefully curated benchmark with expert-authored weighted rubrics and multiple human-scored responses per prompt, and for finding the paper clearly organized with comprehensive experiments. We address each point below; all new results are marked in purple in the revised manuscript.
The key idea of SIGHT is to reuse how human experts build rubrics, rather than what their rubrics say. Existing rubric generators either write rubrics from scratch or retrieve human rubrics of related queries as in-context examples, as in RubricRAG. Retrieval transfers rubric content (the objects, values, and answer requirements of another question) which often do not apply to a new prompt.
SIGHT instead transfers the construction logic: how an expert splits requirements into independently checkable items, separates core from secondary requirements, assigns credits, and defines deductions. To make this logic reusable, we rewrite each human rubric as a construction recipe with instance-specific content removed, so that similarity reflects how a rubric is built rather than what it is about. For each new instance we then derive an evaluation blueprint of its requirements and errors, and select skills to cover that blueprint. The components are familiar, but the unit of reuse is not.
The paper focuses on reward construction for RL. Rubrics serve as reward functions: MM-RubricBench measures whether a reward agrees with human judgment, and SIGHT produces such rewards at scale. Used as RL rewards, SIGHT rubrics improve the base policy by 5.7, 4.3, and 8.0 points on multimodal reasoning, instruction following, and agentic rewarding, outperforming the other reward schemes in each setting. We place the work in reinforcement learning because reward design is central to that setting, and both the benchmark and the method are developed for this use.
Agreement between rubric-conditioned scores and human ratings is useful but indirect, so we also evaluate the rubrics directly. Three human annotators independently compare the rubrics of OpenRubrics, RubricRAG and SIGHT on all 305 prompts of MM-RubricBench, shown anonymously and in random order together with the image, question and reference answer (new Appendix C.5, Figure C.1).
Aspects. For each prompt they choose the best rubric on coverage (every core requirement has a scoring item, and no item asks for something the question does not), correctness (criteria and background knowledge agree with the image and the reference), and usability (credit is placed on the core requirements, equally correct answers can still earn it, and each item can be decided unambiguously). They also mark every rubric that misses or misinterprets a key requirement. Labels follow the majority vote, and prompts on which all three annotators disagree are settled by discussion.
Results. SIGHT is preferred most often on all three aspects: 116 prompts for coverage and 115 for correctness, against at most 102 and 98 for the two baselines. The margin is largest on usability, where SIGHT is chosen on 131 prompts (43.0%) against 92 and 82. Only 5 SIGHT rubrics (1.6%) miss or misinterpret a key requirement, compared with 12 (3.9%) for RubricRAG and 19 (6.2%) for OpenRubrics. We also review 600 rubrics sampled from the three RL training sets and find about 97% correct (new Appendix C.6).