acceptodds
Under review as a conference paper at ICLR 2027

Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation

Abstract

Rubric-based evaluation is widely used to assess LLM-based systems by decomposing response quality into task-specific scoring criteria. However, automatically generating rubrics that reliably capture task-specific quality requirements remains challenging. We introduce Mubric, a mutation testing-guided approach to rubric generation. Mutation testing, a classic software testing methodology, evaluates a test suite by injecting faults into programs and checking whether the tests detect them. We draw an analogy between test suites and rubrics: if a rubric captures an important quality requirement, introducing a corresponding defect into an otherwise high-quality response should reduce its score. Mubric first mines common defects from real pairs of preferred and dispreferred responses and abstracts these defects into reusable mutation operators, each specifying how to introduce a particular type of response defect. For a new task, it applies relevant operators to a reference response, checks whether the injected defects reduce response quality, and uses insufficiently penalized defects to refine the rubric. We evaluate Mubric on 703 tasks across four representative domains against six advanced rubric generation methods. Mubric achieves the highest overall evaluation accuracy, outperforming the strongest baseline by 7.48 percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Rubrics, Mutation Testing, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_LwjTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mubric, a mutation testing-guided framework for automatic rubric generation in LLM evaluation. It constructs reusable mutation operators from real response defects, injects errors into reference responses, and refines rubrics based on their ability to detect these errors. Experiments on 703 tasks across four domains show a 7.48 percentage-point improvement over the strongest baseline.

Critical strengths
  1. Improving automatically generated rubrics is important for scalable LLM evaluation. Introducing mutation testing provides an intuitive way to identify missing evaluation criteria and guide rubric refinement.

  2. The paper evaluates the method across multiple domains and LLM judges, with improvements over several baselines. The ablation studies also provide some evidence supporting the proposed design.

Critical weaknesses
  1. Limited methodological novelty and insufficiently controlled comparisons. The core approach mainly combines existing LLM-based operations with the mutation testing concept. The technical novelty beyond this adaptation appears limited. More importantly, Mubric uses an additional stronger LLM (GPT-5.4) and multiple extra calls, while the baselines do not receive equivalent resources. Without cost-matched comparisons, it is difficult to attribute the improvements specifically to mutation testing.

  2. Limited evidence of practical effectiveness. The evaluation relies primarily on pairwise preference accuracy, which reaches only 57.27% overall. Moreover, the paper does not compare the generated rubrics with expert-written rubrics or sufficiently examine their practical reliability and generation cost. Therefore, the broader value of the method for real-world LLM evaluation remains unclear.

Questions for authors
  1. Can the authors provide comparisons under the same LLM backbone and computational budget? How much improvement can be attributed specifically to mutation testing rather than stronger auxiliary models or additional LLM calls?

  2. How does Mubric compare with expert-written rubrics? Given the relatively low overall accuracy, what evidence supports its reliability and practical value for LLM evaluation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer LwjT

Thanks for your constructive feedback. We have addressed all your questions as follows.

(Question 1) Can the authors provide comparisons under the same LLM backbone and computational budget? How much improvement can be attributed specifically to mutation testing rather than stronger auxiliary models or additional LLM calls?

Thank you for raising this concern. We have conducted an additional experiment to control for both the LLM backbone and inference-call budget simultaneously. Specifically, (1) we used Gemini-2.5-Flash-Lite for all LLM components of Mubric, matching the backbone used by MRRG, the strongest baseline; and (2) we increased MRRG's rubric sampling budget to match Mubric's average LLM call count.

Under this controlled setting, Mubric achieves 50.36% accuracy with 101.6 LLM calls per task, compared with 47.06% for MRRG with 102.0 calls. Thus, Mubric maintains a 3.30-percentage-point advantage despite using the same LLM backbone and a comparable number of calls.

These results demonstrate that Mubric's advantage persists without stronger auxiliary models or additional LLM calls, providing further evidence for the effectiveness of mutation-guided refinement.

(Question 2) How does Mubric compare with expert-written rubrics? Given the relatively low overall accuracy, what evidence supports its reliability and practical value for LLM evaluation?

Thanks for raising this point. Following your suggestion, we have conducted an additional comparison with expert-written rubrics and a paired error analysis on the 574 RubricBench tasks for which expert-written rubrics are available.

Expert-written rubrics achieve 82.23% overall accuracy, outperforming Mubric. However, the paired analysis reveals complementary strengths: Mubric correctly judged 35 of the 102 tasks (34.3%) missed by expert-rubric evaluation. This suggests that Mubric can capture useful evaluation signals missed by expert-rubric-based evaluation, despite its lower overall accuracy.

Mubric's practical value lies in automatically generating task-specific rubrics without requiring labor-intensive, domain-expert rubric authoring. Mubric achieves 57.27% overall accuracy, outperforming the strongest automated baseline (49.78%) by 7.48 percentage points. These findings support Mubric as an automated, complementary alternative for rubric-based LLM evaluation, particularly when expert rubric authoring is impractical.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The new results provide useful evidence, particularly regarding the fairness of the comparisons. However, the substantial performance gap (82.23% for expert rubrics) still raises concerns about the reliability of automatically generated rubrics.

AuthorsOpenReview ↗

Response to Follow-Up by Reviewer LwjT

Thank you for acknowledging the value of our additional experiments.

We agree that expert-written rubrics achieve higher accuracy, and we do not claim that Mubric has reached expert-level reliability. However, closing the gap between automated and expert-written rubrics is precisely a motivation for this line of research, rather than a prerequisite for demonstrating meaningful progress.

Importantly, expert-written rubrics are not always available or practical to obtain, particularly for new tasks or large-scale evaluations. Mubric addresses this need by automatically generating task-specific rubrics, improving overall accuracy from 49.78% (the strongest automated baseline) to 57.27%, without requiring expert-written rubrics. Even when expert rubrics are available, our paired analysis shows that Mubric correctly evaluates 35 of the 102 cases (34.3%) missed by expert-rubric evaluation, suggesting that it can provide complementary evaluation signals.

The remaining gap to expert performance is an important direction for future research, but should not, by itself, negate the demonstrated advances and practical value of automated rubric generation.

Reviewer_6QgzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mubric, a mutation testing-guided approach for automatic rubric generation, which first constructs reusable mutation operators from defects observed in real pairs of preferred and dispreferred data, and then uses these operators to probe and refine task-specific rubrics. They conduct experiments to evaluate Mubrics on 703 tasks across four domains and show that the proposed method achieves the highest overall evaluation accuracy.

Critical strengths
  • This paper aims at an important problem in rubrics-based LLM evaluation, and the mutation-testing based method provides an intuitive way to identify and refine rubrics that can be missing.
  • The evaluation tasks have high coverage across different domains, which proves the generalization capability of the proposed method.
  • The evaluation results are promising, with rich ablation studies to provide insights into how each component of Mubric contributes to the improvement.
Critical weaknesses
  • Though the mutation-test guided refinement method is intuitive, it is unclear to me how much different mechanism it can provide comparing to the existing adversarial negative generation based method. In particular, I found it shares similar ideas with "Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics".

  • This method is based on the assumption that the injected defects are valid and that increasing the rubric's sensitivity to these defects improves evaluation quality. However, it remains unclear how trustworthy these defects are, as in this paper the mutation is primarily verified by the LLM-based verification. Additional experiments on independently validated mutations and quality-preserving transformations would help to understand whether this method improves general rubric reliability rather than the sensitivity to a particular type of synthetic defects.

Questions for authors
  1. How frequently do mutation operators produce valid defects v.s. discarded defects. And how is this validated?
  2. How does the proposed method affect the inference cost overhead?
  3. In Table 2, the initial baseline in the ablation study performs surprisingly well, outperforming several existing methods in Table 1 despite its simple design. Could the authors explain what might account for this strong performance? In particular, how much of Mubric's improvement over baselines comes from the initial rubric generation v.s. the proposed mutation-guided refinement?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6Qgz [1/2]

Thanks for your constructive feedback. We have addressed all your questions as follows.

(Question 1) How frequently do mutation operators produce valid defects v.s. discarded defects. And how is this validated?

Of 3,646 generated mutations, 89.7% (3,269) passed LLM-based verification, while 10.3% (377) were discarded.

To assess the reliability of LLM-based verification, we conducted a manual audit. We randomly sampled 348 mutated responses from all 3,646 generated mutations, including both accepted and discarded ones. The sample size was determined using a 95% confidence level and a ±5 percentage-point margin of error, providing a statistically grounded basis for the assessment.

Two annotators experienced in AI independently examined each mutated response, along with its task description, reference response, and mutation operator, without knowing the LLM verifier's decision. They assessed whether the intended defect was successfully introduced, whether it was introduced without unrelated defects, whether the mutation genuinely reduced response quality for the given task, and whether its effects were confined to the intended quality aspect. Their initial agreement was 95.98%. Disagreements were resolved through discussion, with a third annotator adjudicating any unresolved cases. This process produced an independent human validity label for each sampled mutation.

We then compared the LLM verifier's decisions against these human labels. The verifier achieved 97.78% precision and 93.90% recall. These results provide strong independent evidence that our LLM-based verifier reliably identifies valid mutations and that the retained mutations used for rubric refinement predominantly represent genuine quality-degrading defects.

(Question 2) How does the proposed method affect the inference cost overhead?

Thank you for the question. We compare the full-pipeline inference cost of Mubric with MRRG, the strongest competing baseline, across both LLM backbones and all five datasets. Mubric requires 99.59 LLM calls and \0.25pertask,comparedwith73.25callsand0.25 per task, compared with 73.25 calls and \\0.04 for MRRG. The additional overhead primarily arises from mutation generation, verification, and rubric refinement.

Importantly, mutation-guided refinement is performed during rubric construction rather than each subsequent response evaluation. Once generated, the rubric can be reused without repeating mutation testing. Thus, Mubric trades higher inference cost for improved evaluation accuracy (57.27% vs. 49.78% for MRRG), while allowing the additional construction cost to be amortized across repeated evaluations.

(Question 3) In Table 2, the initial baseline in the ablation study performs surprisingly well, outperforming several existing methods in Table 1 despite its simple design. Could the authors explain what might account for this strong performance? In particular, how much of Mubric's improvement over baselines comes from the initial rubric generation v.s. the proposed mutation-guided refinement?

We hypothesize that Initial's strong performance partly stems from its scoring protocol: evaluating each criterion independently, allowing partial credit on a 0–4 scale, and weighting criteria by task importance.

To investigate this, we have conducted additional experiments varying the scoring granularity (joint vs. itemized), scoring scale (binary vs. 0–4), and criterion weights (equal vs. importance-based), while keeping the initial rubrics and evaluation samples fixed. Itemized scoring improves accuracy from 37.32% to 41.05%, and importance weighting further improves it from 38.57% to 41.05%. Notably, the 0–4 scale increases accuracy from 41.05% to 47.94%. These findings suggest that the scoring protocol may partly explain Initial's strong performance relative to several existing baselines, despite its simple rubric generation process.

Crucially, mutation-guided refinement provides a further, statistically significant improvement. Under the same scoring protocol, Mubric improves accuracy from 47.94% (Initial) to 51.78% (+3.84 percentage points; paired exact McNemar test, p=4.19×10−5p = 4.19 \times 10^{-5}). This demonstrates that Mubric's gains are not solely attributable to strong initial rubric generation, but also to the proposed mutation-guided refinement.

AuthorsOpenReview ↗

Response to Reviewer 6Qgz [2/2]

How much different mechanism it can provide comparing to the existing adversarial negative generation based method. In particular, it shares similar ideas with "Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics".

Thank you for highlighting Support Vector Rubrics (SVR). Although both methods leverage negative responses, their underlying mechanisms differ:

(1) Training-based vs. training-free. SVR learns a rubric selector and global weights through max-margin optimization, whereas Mubric requires no model training.

(2) Adversarial optimization vs. mutation testing. SVR uses adversarial hard negatives to refine discriminative boundaries. In contrast, Mubric constructs reusable mutation operators from real response defects and uses them to systematically test whether a rubric detects specific quality violations. Only verified, insufficiently penalized defects trigger targeted refinement, which is subsequently retested.

The key distinction is that SVR focuses on learning discriminative boundaries, whereas Mubric introduces a training-free, defect-driven testing and repair mechanism for task-specific rubrics.

Thanks for the detailed responses and additional experiments. The human validation of mutation quality provides convincing evidence for the reliability of the verification process, and the new ablation studies help clarify the gap between the initial ablation baseline and the public baselines.

However, my main concerns regarding the novelty and significance of the proposed method remain. The additional experiments show that relatively simple changes to the scoring protocol can lead to substantial improvements, in some cases exceeding the incremental gain from mutation-guided refinement itself. Therefore, it remains unclear to what extent Mubric's advantage over existing approaches would persist under a unified scoring protocol and comparable inference budgets.

Thank you for your encouraging feedback and for highlighting this remaining concern.

To directly address this concern, we conducted an additional experiment that simultaneously controls for the scoring protocol and inference-call budget. Specifically, we applied Mubric's scoring protocol to MRRG, the strongest baseline, used Gemini-2.5-Flash-Lite for all LLM components, and increased MRRG's rubric sampling budget to match Mubric's inference-call budget.

Under these controlled conditions, Mubric achieves 50.36% accuracy with 101.6 calls per task, compared with 45.95% for MRRG with 104.1 calls. This 4.41-percentage-point advantage is statistically significant (p < 0.05).

Importantly, MRRG uses slightly more inference calls under the same scoring protocol and LLM backbone, yet Mubric still achieves significantly higher accuracy. These results demonstrate that Mubric's advantage cannot be explained solely by scoring-protocol differences or additional inference calls, providing direct evidence of the added value of mutation-guided refinement.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Response to Reviewer guk6 [1/3]

Thank you for your constructive feedback. We have addressed all your questions as follows.

(Question 1) Can you run Mubric with all auxiliary components using the same GPT-5.4-mini / Gemini-2.5-flash-lite backbone as the baselines? How much of the +7.48-point gain remains?

Following your suggestion, we have conducted an additional experiment using Gemini-2.5-Flash-Lite for all LLM components of Mubric, matching the backbone used by MRRG, the strongest baseline.

Under this unified-backbone setting, Mubric achieves 50.36% overall accuracy, outperforming MRRG (45.38%) by 4.98 percentage points. This demonstrates that a substantial performance advantage remains even without stronger auxiliary models, providing evidence that Mubric's improvements are not solely attributable to their use.

(Question 2) How many additional LLM calls, tokens, and dollars does Mubric require per task compared with MRRG, Dynamic, and RLCF?

Thank you for the question. We have measured Mubric's overhead relative to MRRG, Dynamic, and all three RLCF variants across both LLM backbones and all five datasets. The costs include LLM calls, input/output tokens, and token pricing.

Additional inference cost of Mubric per task relative to each baseline (GPT-5.4-mini / Gemini-2.5-Flash-Lite)

BaselineAdditional callsAdditional input tokens (k)Additional output tokens (k)Additional cost (USD)
MRRG17.22 / 34.77295.56 / 42.814.99 / 2.930.32 / 0.11
Dynamic100.32 / 95.18362.38 / 102.177.79 / 5.790.39 / 0.12
RLCF-R74.61 / 77.12341.67 / 86.686.84 / 4.910.35 / 0.10
RLCF-B75.72 / 74.02341.50 / 83.704.53 / 2.420.34 / 0.10
RLCF-C79.36 / 76.75344.82 / 86.245.16 / 3.100.36 / 0.11

The additional overhead primarily arises from mutation generation, verification, and rubric refinement. These operations are performed during rubric construction rather than each subsequent response evaluation. Once generated, a rubric can be reused without repeating mutation testing, allowing the additional construction cost to be amortized across repeated evaluations.

(Question 3) On average, how many operators are selected per task, how many mutations pass verification, how many trigger refinement, and how many proposed refinements are retained?

On average, 2.59 operators are selected per task, 2.33 mutations pass verification, 0.82 trigger refinement, and 95.0% of the proposed refinements are retained.

(Question 4) Have you evaluated refined rubrics on held-out mutation operators that were not used to generate the refinement?

Following your suggestion, we have conducted an additional held-out mutation evaluation using Gemini-2.5-Flash-Lite across all five datasets.

For each task, we separated the mutation operators used for rubric refinement from held-out operators. We then applied the held-out operators to generate new mutated responses and compared how frequently the initial and refined rubrics sufficiently penalized these mutations. The held-out mutations were not used to guide refinement or retesting.

The results show that the detection rate on held-out mutations improves from 39.72% to 42.99% (+3.27 percentage points), with improvements across all five datasets. These findings provide direct evidence that mutation-guided refinement improves the detection of unseen defects, rather than merely fitting the mutations used during refinement.

AuthorsOpenReview ↗

Response to Reviewer guk6 [2/3]

(Question 5) Can you test the refined rubrics against human-written incorrect responses or defects from mutation families excluded during refinement?

Thanks. We have further extended the held-out operator evaluation described in our response to Question 4 to a more challenging setting by excluding entire mutation families from refinement, rather than individual operators.

The detection rate on these held-out families improves from 37.71% to 40.77% (+3.07 percentage points), with improvements across all five datasets.

These results extend our previous findings from unseen operators to unseen defect families, providing further evidence that mutation-guided refinement is not limited to the specific defect types used during refinement.

(Question 6) How was mutation validity checked? Do you have a human audit measuring whether mutations?

Thank you for highlighting this concern. To assess the reliability of LLM-based mutation verification, we have conducted a human audit of 348 randomly sampled mutated responses from all 3,646 generated mutations, including both accepted and discarded ones. The sample size was determined using a 95% confidence level and a ±5 percentage-point margin of error.

Two annotators experienced in AI independently examined each mutation alongside its task description, reference response, and mutation operator, without knowing the LLM verifier's decision. They assessed whether the intended defect was introduced, unrelated content and quality were preserved, response quality genuinely decreased, and the degradation was confined to the intended quality aspect. A mutation was labeled valid only if all four conditions were satisfied. Initial inter-annotator agreement was 95.98%, with disagreements resolved through discussion or third-party adjudication.

Comparing the verifier's decisions against the final human labels, we obtained 97.78% precision and 93.90% recall. These results provide strong independent evidence that the LLM-based verifier reliably filters mutations and that the retained mutations predominantly represent valid, isolated, quality-degrading defects.

(Question 7) What fraction of generated reference responses contain errors? What happens if the benchmark's preferred response is used as the mutation anchor instead of an independently generated reference?

(1) Reference response quality: We have conducted an independent human audit of 240 randomly sampled reference responses from the responses that passed our initial quality check. The sample size was determined using a 95% confidence level and a ±5 percentage-point margin of error, providing a statistically grounded basis for the audit. Two annotators independently examined each task description and reference response for errors, without access to other responses. Their agreement was 98.33%, with disagreements resolved through discussion or third-party adjudication. Only 2.5% of the audited reference responses contained errors, providing evidence that the retained reference responses are generally of high quality.

(2) Alternative mutation anchors: Using benchmark-preferred responses as mutation anchors requires access to ground-truth preference labels, which are unavailable for new tasks and are precisely what the resulting rubrics are evaluated against. This introduces evaluation-label information into rubric construction, departing from Mubric's intended label-free setting.

Nevertheless, we have conducted the suggested experiment across all five datasets using Gemini-2.5-Flash-Lite and otherwise identical pipeline settings. Using benchmark-preferred responses yields 42.73% overall accuracy, compared with 51.78% using independently generated references (−9.05 percentage points). These results support our choice of independently generated references, both for maintaining label-independent rubric construction and for achieving better empirical performance.

AuthorsOpenReview ↗

Response to Reviewer guk6 [3/3]

(Question 8) In the threshold sweep, how many mutation decisions and refinement decisions actually change across the seven settings? Are the final rubrics textually different?

Thanks. We have further analyzed how the seven threshold settings affect mutation detection decisions, refinement decisions, and final rubric texts. The table reports the proportions of changes relative to the default setting (δtotal=8,δitem=2\delta_{total}=8, \delta_{item}=2).

Thresholds (δtotal\delta_{total}, δitem\delta_{item})Proportions of changes mutation decisionsProportions of changed refinement decisionsProportions of textually changed rubrics
(4, 2)2.88%1.85%8.33%
(12, 2)0.79%1.85%2.78%
(16, 2)0.79%1.85%2.78%
(8, 1)6.81%6.48%19.44%
(8, 3)4.71%4.63%13.89%
(8, 4)4.71%4.63%17.59%

Across the six alternative settings, mutation detection decisions change by 0.79%-6.81%, refinement decisions by 1.85%-6.48%, and 2.78%-19.44% of final rubrics are textually different. However, as shown in Appendix B (Figure 3), these variations have little impact on final evaluation accuracy, indicating that Mubric's evaluation performance is robust to threshold choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.