acceptodds
Withdrawn from ICLR 2027

From Cases to Criteria: Reusing Evaluation Knowledge for Task-Specific Rubric Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Query-specific rubrics make LLM evaluation more fine-grained and transparent, and generating high-quality rubrics requires both identifying the target query’s evaluation scope and specifying concrete checks. Historical rubrics provide reusable evaluation experiences, yet existing approaches reuse either complete query–rubric cases or aggregated criteria and dimensions without explicitly connecting recurring evaluation requirements to their criterion-level evidence and source-query context. This can preserve irrelevant task details or lose the broader evaluation structure. In this paper, we introduce RubricDuo, which organizes historical query–rubric pairs into a Rubric Knowledge Graph linking evaluation dimensions, shared concepts, local concepts, criteria, and queries, where reusable evaluation requirements across queries are abstracted by shared concepts. Given a target query, RubricDuo first combines concepts relevant to the target query with retrieved historical cases to draft a rubric with broad coverage. It then revisits each draft criterion, retrieves concept-linked historical criteria and their source queries, and adapts applicable evidence to refine the criterion. Experiments on RubricBench and HealthBench demonstrate that RubricDuo achieves the overall best performance among the evaluated automatic methods in both alignment with human-authored rubrics and downstream LLM judging.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM evaluation, Rubric Generation, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZCVAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RubricDuo, which organizes historical query–rubric pairs into a knowledge graph connecting evaluation concepts with their source criteria and queries. It generates task-specific rubrics through concept-guided drafting followed by criterion-level refinement, improving rubric alignment and downstream judging on RubricBench and HealthBench.

Critical strengths

The method is clearly presented, with an understandable knowledge organization scheme and a well-defined draft–refine pipeline.

Critical weaknesses
  • Insufficient component-level validation. Table 3 compares graph-guided and flat retrieval, but does not isolate the contributions of concept prompting, concept-based filtering, and source-query context. Consequently, the benefit of the proposed multi-level graph organization remains insufficiently established. Table 4 only evaluates refinement as a whole.

  • Sec. 3 requires offline graph construction and online concept matching, retrieval, and criterion-level refinement, yet their costs are not reported. Given the modest refinement gains in Table 4, the quality–cost trade-off is difficult to assess.

Questions for authors
  1. Table 5 shows that with Qwen3.8-max, RubricDuo improves rubric F1 but does not outperform RubricRAG in downstream accuracy. What explains this discrepancy, and do the gains persist with stronger judges or judges from another model family?

  2. Appendix A.1 uses an LLM to match generated and human-authored criteria. Which model performs this matching, and has its reliability been checked against human judgments?

  3. How sensitive is shared-concept induction to the order of historical cases and the initialization batch? Can the authors provide examples of successful cross-topic transfer and failures caused by inappropriate concept matching?

Reviewer_mmzeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses historical scoring criteria to help the model generate scoring criteria for new problems. RubricDuo organizes historical issues and standards into a knowledge map, connecting related requirements through shared concepts. It first drafts based on relevant concepts and retrieved cases, then gathers more evidence and revises item by point. Experiments on RubricBench and HealthBench examined two things: whether the generated standards matched the human standard, and whether these standards could help LLM judges select better answers.

Critical strengths
  1. The problem is worth studying. Writing a good rubric for every query takes effort. Reusing earlier rubrics makes sense, but copying details from a similar query can go wrong. Keeping the original query alongside a criterion is a sensible design choice. The shared concepts also give the method a way to find useful criteria from different topics.
  2. There are useful gains and relevant comparisons. On RubricBench, judging accuracy reaches 55.56%, compared with 49.13% for Flat-Hybrid and 45.51% for Direct-Q. I appreciate that Table 3 includes flat retrieval methods with the same two-stage structure. The paper also checks both rubric alignment and downstream judging, and includes cross-benchmark transfer and different generators.
Critical weaknesses
  1. I am not yet convinced which part of the workflow makes the difference. The method adds concept labels, filtering, source context, and several generation steps. Table 3 helps, but it is not clear whether the flat baselines get the same concept information, source queries, and generation budget. Could a flat index with the same labels and source context work as well? Could a general retrieval-and-revision workflow get similar results with the same budget? These comparisons would help establish what this particular design adds.

  2. It is unclear what happens when there is no useful history to retrieve. The method retrieves the top three cases and top six criteria. If concept filtering finds nothing, it falls back to the corresponding dimensions. This can still return examples that are not useful for the query. I could not find a relevance threshold or a clear rule for proceeding without retrieved evidence. This matters because avoiding the transfer of irrelevant details is part of the paper's motivation. The cross-benchmark experiment is useful, but it does not show how the method behaves specifically on queries with little relevant history. I would like to know whether retrieval can hurt these queries and whether Direct-Q would be a better choice for them.

Questions for authors
  1. What exactly is held fixed in Table 3? Do the flat baselines receive the same concept information and source-query context, and use the same retrieval counts and generation budget?

  2. When the retrieved material is unsuitable, can the generator ignore all of it? Do you have results split by retrieval relevance or historical coverage, including a comparison with Direct-Q on the poorly supported queries?

  3. How much does graph construction cost, and how many calls or tokens does each new query require? Can you provide paired uncertainty estimates for the gains over the strongest baselines and for the 1.93-point gain from refinement in Table 4?

  4. The related work misses TestAgent (Wang et al., ICASSP 2026), which also generates evaluation criteria in two stages, first at the topic level and then at the question level using retrieved evidence. The authors should discuss this connection and clarify what the historical rubric graph and shared concepts add beyond this earlier approach. A comparison with an adapted version of its criteria-generation component would help, with differences in available inputs made explicit.

Reviewer_3DZnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RubricDuo generates query-specific evaluation rubrics for LLM-as-a-judge by reusing historical human-written query–rubric pairs at the level of "shared evaluation concepts" rather than whole cases or raw criteria. Offline, an LLM annotates each historical criterion with one of six dimensions (Domain, Task, Response Protocol, Constraint, General Quality, Safety & Policy), induces a local concept per dimension, and merges local concepts into shared concepts (490 for RubricBench, 335 for HealthBench), producing a five-level "Rubric Knowledge Graph" that links dimensions → shared concepts → local concepts → criteria → queries. Online, Step I selects shared concepts relevant to the target query and retrieves top-3 similar cases (dense + BM25 with RRF) to draft a rubric; Step II revisits each draft criterion, filters historical criteria by the draft criterion's concepts, retrieves the top-6 by hybrid search, and asks the LLM to refine the criterion using them and their source queries. With Qwen3.7-flash as generator, matcher, weight assigner and judge, RubricDuo reports the best structure-F1 against held-out human rubrics on RubricBench (50.72 vs 48.92 for the best baseline) and HealthBench (46.89 vs 44.77), the best downstream pairwise accuracy on RubricBench (55.56% vs 52.22% for CaRG; human rubrics 72.61%), and the best good-over-bad accuracy on HealthBench (49.33 vs 48.37). Ablations compare flat versus graph-guided retrieval, draft versus refined rubrics, three generator sizes, and cross-benchmark transfer.

Critical strengths
  1. The concept-level provenance design is sensible and tested against the right alternatives. Reusing a budget-compliance requirement from a restaurant rubric for a hotel query is a real failure of similarity-based reuse, and the baseline set covers query-only, random demonstrations, similar cases, similar criteria and dimension pools under a shared generator, corpus and judge. The cross-generator result, where structure lets a 14B generator reach the alignment of larger-model baselines, is the most practically useful finding.
Critical weaknesses
  1. The improvements sit within run-to-run and sampling noise, and the central ablation does not show what it claims. On HealthBench, 49.33 plus or minus 1.81 overlaps Rubric-Ret at 48.37 plus or minus 1.34 and RubricRAG at 47.62 plus or minus 2.06; on RubricBench the 3.3-point gap over CaRG is computed on 230 queries, whose binomial standard error alone is about 3.3 points, with no significance test reported. Section 4.4 claims that flat retrieval "reduces accuracy by more than six points", but Table 4 shows the refinement step, the only place graph-guided criterion retrieval acts, adding 1.93 accuracy over the draft, and draft-only RubricDuo already reaches 53.63. Most of the six-point gap comes from concept selection during drafting, not from the graph, and the 2x2 separating the two components is missing.

  2. One model acts as generator, matcher, weight assigner and judge, with no human validation anywhere. Qwen3.7-flash builds the graph, writes the rubrics, performs the criterion matching that defines precision, recall and F1, assigns HealthBench weights and renders the downstream verdicts. Agreement between that matcher and human matching is never measured, so "alignment with human-authored rubrics" is one model's opinion of similarity to human rubrics. On HealthBench even human rubrics reach only 55.56% good-over-bad accuracy and No-Rubric is not reported, so it is unclear whether any generated rubric helps there at all.

  3. The baselines are not cost-matched, and the prompts used to run three of them carry a corrupted instruction. RubricDuo drafts a rubric and then issues a separate concept-filtered retrieval and refinement call for every draft criterion, roughly an order of magnitude more generator calls per query than Direct-Q or RubricRAG, and no cost-matched control such as Direct-Q with self-refinement is reported, so the gain may be bought with compute rather than with knowledge reuse. Separately, the prompts in Appendix A.3 for RubricRAG, Random-FS and Rubric-Ret instruct the generator that each criterion must be "criterionic", evidently a find-and-replace failure on "atomic". If the reported baselines were run with that text, they are not a fair reference.

Questions for authors
  1. Please report paired significance (e.g. bootstrap over test queries, or McNemar for accuracy) for RubricDuo versus the best baseline on each metric, and the 2×2 ablation separating concept-guided drafting from graph-guided refinement.

  2. What is the agreement between the LLM structure matcher and human matching on a sample of generated–human criterion pairs? Does the ranking of methods change under a different matcher model (e.g. a non-Qwen model)?

  3. What is No-Rubric G>B accuracy on HealthBench, and why are all weighted-rubric accuracies, including human rubrics, at or below 56%? Are ties frequent?

  4. The baseline prompts in A.3 contain the word "criterionic". Were the reported baselines run with this text? Could it have affected their outputs?

  5. How many LLM calls and tokens does RubricDuo use per query compared with Direct-Q, RubricRAG and CaRG? The per-criterion refinement implies roughly an order of magnitude more calls; a cost-matched comparison (e.g. Direct-Q with self-refinement) would show whether the gain is from knowledge reuse or from extra compute.

  6. RubricBench includes safety tasks and the graph has a Safety & Policy dimension; a per-category breakdown would show where concept reuse helps most.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.