acceptodds
Under review as a conference paper at ICLR 2027

RubricCraft: Where Solutions Diverge, Criteria Emerge

Abstract

Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical. Textual rubrics make evaluation criteria explicit, but writing them requires domain expertise. Rubrics generated from task descriptions alone can miss requirements and failure modes that become apparent in attempted solutions. We introduce RubricCraft, which treats rubric construction as the generation of natural-language specifications guided by candidate solutions. Given a task description and independently sampled solver trajectories, RubricCraft compares how each attempt satisfies or violates the task requirements and uses these differences to construct rubric criteria. It applies the criteria to the collected solutions and revises checks that overlook errors or penalize valid alternatives. The resulting rubrics capture partial correctness without requiring any single solution to serve as a complete reference. On SpreadsheetBench v2, against a prompt-only rubric written and applied by the same model, RubricCraft rubrics pass fewer wrong outputs (29.5% against 42.6%), fail none of the outputs the benchmark's own checker accepts, and rank outputs by correctness more often, while separating models as often. Repair on a single attempt leaves grading unchanged, but repair driven by attempts from three model families acts on different tasks, and an evidence audit judges most of those repairs improvements, many on tasks whose reference solution is itself wrong. We release the induced rubrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, agent evaluation, rubric generation, rubric-based evaluation, natural-language specification generation, benchmarking, evaluation robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r2q2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of defining rubrics for evaluation of open-ended LLM tasks where the output is an artifact. They use a multi-stage LLM pipeline where some LLMs attempt the task, some LLMs generate rubrics, and then different attempts are checked against each other and against the rubrics to find inconsistencies between them and repair the criteria. The paper classifies defective criteria based on their redundancy (cannot distinguish attempts), vagueness (the same attempt may get different verdicts), incompleteness (missing a requirement) and over-restrictiveness (called unfairness). The pipeline is set up and tested on a SpreadsheetBench, a benchmark about spreadsheet editing.

Critical strengths

The paper adapts an established technique from specification validation to LLM artifact evaluation, namely rubric refinement over independent attempts, tests, and counterexample-guided repair loops.

The related work is very thorough and informative.

Critical weaknesses

The framework is only tested on one benchmark. Further, while the authors claim that it generalizes, many of their definitions are specific to this benchmark (for example, cell accuracy). So I do not see how this can generalize to other settings.

The writing is very hard to follow. Metrics, methodology, definitions and experimental details are not separated. Many choices are underexplained, many terms are not formally defined, and many unimportant implementation details are written in the main (for example, the exact commit of the dataset used).

Questions for authors

Questions:

  • You claim that a criterion is incomplete if attempts differ on some axis that is not addressed. But is this necessarily a problem? Can't there be multiple different correct solutions?

  • How would rubriccraft work on a benchmark without a golden workbook?

  • How do you decide on the initial configuration of models and parameters used for attempts?

  • How do you get the initial rubric R_0 ? How does it differ from naive?

  • You claim you found errors in the benchmark's evaluator. Can you explain these errors? Perhaps in an Appendix.

  • In section 5, I am completely confused on how many tasks you considered. Initially, you say you produced a rubric for 302, 278 nonvisual. Then you say there are 259 nonvisual. Then you compare values on 242 tasks. Then you say the full task number is 321. Can you please clarify?

  • There are many different "accuracy" metrics and it is unclear to me how they are used. Percentage score is [0,100]. Judge grades are [0,4] then multiplied by 25. Seed grade is [0,1] multiplied by 100. Cell accuracy in [0,1] compares with an expert-written "golden workbook", and an output is considered wrong if this is less than 0.8 (why 0.8?). But on line 287 an output is correct if every cell is correct and 99.8% of preserved cells are unchanged (why 99.8?). And later on line 300 an output is wrong if fewer than 80% match, and a rubric fails an output at below 3/4, why? Can you clarify these choices and definitions?

Suggestion for revision:

  • First define the defects generally and then write about how you fix them in your repair loop. Also, define your framework for general benchmarks. Currently, all of this is muddled up together and many things are defined specifically for SpreadsheetBench (for example, cell accuracy).
Reviewer_w2vcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for generation of rubrics to evaluate solutions to artifact generation tasks by leveraging the natural language specifications and candidate solutions. Using these solutions, the method identifies rubric defects like missing criteria (by comparing differences across candidate solutions), non-discriminative criteria, non-checkable criteria and unfair criteria (by seeing if a valid solution is rejected). The defects are iteratively repaired and re-evaluated on SpreadsheetBenchv2 which requires manipulation of data to achieve required results.

Critical strengths

S1: The problem is crucial and timely since rubric-based training is becoming an increasingly popular paradigm. Improving their reliability and adapting them to new tasks are important directions.

S2: I like the core idea that observable differences in solutions can drive the rubric generation. The taxonomy of the type of errors can also be useful for future works.

S3: The evaluation across granular metrics like agreement, fidelity, etc. provide a clearer picture than an aggregate score alone.

Critical weaknesses

W1: The main experiments do not convincingly establish the benefit of the rubric repair. Some results, like no-loop rubric, perform similar to or even better than the final repaired rubric. It is hard to tell if there is a meaningful empirical benefit of the rubric repair process.

W2: Cost and scalability are major concerns since a popular use for rubrics is training. Multiple candidate solutions would be required for each instance. This concern is less important for evaluation settings, as the authors correctly point out.

W3: The criteria are induced solely from the solution attempts which makes shared mistakes or missing behaviors difficult to surface. This can also lead to extra, overly specific criteria to discriminate between solutions.

W4: The evaluation does not show generalizability to other more open-ended tasks than SpreadsheetBench, which has a golden artifact and programmatic checks available. Slide generation can be a useful target.

Questions for authors

Q1: If a rubric is constructed with one set of models, how well does it generalize to held-out ones?

Q2: How does the quality of the final rubrics scale with the number of candidate solutions used to generate it? This will help understand the cost-quality tradeoff.

Reviewer_KkbkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors address the challenge of crafting rubrics for LLM and agentic judges. They propose a general pipeline to generate and improve rubrics using LLMs and apply it to SpreadsheetBenchv2. They then show that the rubrics they create substantially reduce the judges' errors.

Critical strengths
  • As agentic judges are becoming increasingly prevalent, the number of rubrics that will need to be constructed will explode. So a work that helps to stem the tide is quite valuable.
  • The results presented here show a fairly substantial improvement when using this to improve the rubrics and they actually conducted significance tests in their empirical work (a sadly rare but important contribution).
Critical weaknesses
  • I think the biggest weakness is that this work is effectively trying to solve a problem that has been studied for a very long time. Education and psychology have studied how to construct rubrics for a very long time, and a very strong justification is needed to completely ignore this. I would think that, at least, an actual post-hoc analysis of the rubrics in relation to doctrines provided by educational research is needed to be able to actually justify this system. If the rubrics do not match, then a reason needs to be provided as to why.
  • The authors only look at one benchmark here. This raises some questions about how much this will generalize to other benchmarks.
Questions for authors
  • Is it possible to apply or modify this method to work in cases where no clear ground truth exists (i.e., when systems are deployed rather than just benchmarked)? Would the rubrics likely transfer from the benchmarks to real-world problems that agentic AI and LLMs are being applied to?
  • Do the rubrics overly specialize to the individual tasks? Saying yes would not make me any more likely to reject this work, but I would be curious how general the resulting rubrics are.

Please also respond to the points raised in the weaknesses section.

1 comment
Reviewer_KkbkOpenReview ↗

Response to authors

I appreciate the work the authors have put into grounding this work more solidly. I'm not exactly blown away by this still, but that was my major concern so I'll raise my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.