- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Mubric, a mutation testing-guided framework for automatic rubric generation in LLM evaluation. It constructs reusable mutation operators from real response defects, injects errors into reference responses, and refines rubrics based on their ability to detect these errors. Experiments on 703 tasks across four domains show a 7.48 percentage-point improvement over the strongest baseline.
Critical strengths
-
Improving automatically generated rubrics is important for scalable LLM evaluation. Introducing mutation testing provides an intuitive way to identify missing evaluation criteria and guide rubric refinement.
-
The paper evaluates the method across multiple domains and LLM judges, with improvements over several baselines. The ablation studies also provide some evidence supporting the proposed design.
Critical weaknesses
-
Limited methodological novelty and insufficiently controlled comparisons. The core approach mainly combines existing LLM-based operations with the mutation testing concept. The technical novelty beyond this adaptation appears limited. More importantly, Mubric uses an additional stronger LLM (GPT-5.4) and multiple extra calls, while the baselines do not receive equivalent resources. Without cost-matched comparisons, it is difficult to attribute the improvements specifically to mutation testing.
-
Limited evidence of practical effectiveness. The evaluation relies primarily on pairwise preference accuracy, which reaches only 57.27% overall. Moreover, the paper does not compare the generated rubrics with expert-written rubrics or sufficiently examine their practical reliability and generation cost. Therefore, the broader value of the method for real-world LLM evaluation remains unclear.
Questions for authors
-
Can the authors provide comparisons under the same LLM backbone and computational budget? How much improvement can be attributed specifically to mutation testing rather than stronger auxiliary models or additional LLM calls?
-
How does Mubric compare with expert-written rubrics? Given the relatively low overall accuracy, what evidence supports its reliability and practical value for LLM evaluation?
3 comments
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The new results provide useful evidence, particularly regarding the fairness of the comparisons. However, the substantial performance gap (82.23% for expert rubrics) still raises concerns about the reliability of automatically generated rubrics.
Response to Follow-Up by Reviewer LwjT
Thank you for acknowledging the value of our additional experiments.
We agree that expert-written rubrics achieve higher accuracy, and we do not claim that Mubric has reached expert-level reliability. However, closing the gap between automated and expert-written rubrics is precisely a motivation for this line of research, rather than a prerequisite for demonstrating meaningful progress.
Importantly, expert-written rubrics are not always available or practical to obtain, particularly for new tasks or large-scale evaluations. Mubric addresses this need by automatically generating task-specific rubrics, improving overall accuracy from 49.78% (the strongest automated baseline) to 57.27%, without requiring expert-written rubrics. Even when expert rubrics are available, our paired analysis shows that Mubric correctly evaluates 35 of the 102 cases (34.3%) missed by expert-rubric evaluation, suggesting that it can provide complementary evaluation signals.
The remaining gap to expert performance is an important direction for future research, but should not, by itself, negate the demonstrated advances and practical value of automated rubric generation.
Response to Reviewer LwjT
Thanks for your constructive feedback. We have addressed all your questions as follows.
Thank you for raising this concern. We have conducted an additional experiment to control for both the LLM backbone and inference-call budget simultaneously. Specifically, (1) we used Gemini-2.5-Flash-Lite for all LLM components of Mubric, matching the backbone used by MRRG, the strongest baseline; and (2) we increased MRRG's rubric sampling budget to match Mubric's average LLM call count.
Under this controlled setting, Mubric achieves 50.36% accuracy with 101.6 LLM calls per task, compared with 47.06% for MRRG with 102.0 calls. Thus, Mubric maintains a 3.30-percentage-point advantage despite using the same LLM backbone and a comparable number of calls.
These results demonstrate that Mubric's advantage persists without stronger auxiliary models or additional LLM calls, providing further evidence for the effectiveness of mutation-guided refinement.
Thanks for raising this point. Following your suggestion, we have conducted an additional comparison with expert-written rubrics and a paired error analysis on the 574 RubricBench tasks for which expert-written rubrics are available.
Expert-written rubrics achieve 82.23% overall accuracy, outperforming Mubric. However, the paired analysis reveals complementary strengths: Mubric correctly judged 35 of the 102 tasks (34.3%) missed by expert-rubric evaluation. This suggests that Mubric can capture useful evaluation signals missed by expert-rubric-based evaluation, despite its lower overall accuracy.
Mubric's practical value lies in automatically generating task-specific rubrics without requiring labor-intensive, domain-expert rubric authoring. Mubric achieves 57.27% overall accuracy, outperforming the strongest automated baseline (49.78%) by 7.48 percentage points. These findings support Mubric as an automated, complementary alternative for rubric-based LLM evaluation, particularly when expert rubric authoring is impractical.