- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the problem of defining rubrics for evaluation of open-ended LLM tasks where the output is an artifact. They use a multi-stage LLM pipeline where some LLMs attempt the task, some LLMs generate rubrics, and then different attempts are checked against each other and against the rubrics to find inconsistencies between them and repair the criteria. The paper classifies defective criteria based on their redundancy (cannot distinguish attempts), vagueness (the same attempt may get different verdicts), incompleteness (missing a requirement) and over-restrictiveness (called unfairness). The pipeline is set up and tested on a SpreadsheetBench, a benchmark about spreadsheet editing.
Critical strengths
The paper adapts an established technique from specification validation to LLM artifact evaluation, namely rubric refinement over independent attempts, tests, and counterexample-guided repair loops.
The related work is very thorough and informative.
Critical weaknesses
The framework is only tested on one benchmark. Further, while the authors claim that it generalizes, many of their definitions are specific to this benchmark (for example, cell accuracy). So I do not see how this can generalize to other settings.
The writing is very hard to follow. Metrics, methodology, definitions and experimental details are not separated. Many choices are underexplained, many terms are not formally defined, and many unimportant implementation details are written in the main (for example, the exact commit of the dataset used).
Questions for authors
Questions:
-
You claim that a criterion is incomplete if attempts differ on some axis that is not addressed. But is this necessarily a problem? Can't there be multiple different correct solutions?
-
How would rubriccraft work on a benchmark without a golden workbook?
-
How do you decide on the initial configuration of models and parameters used for attempts?
-
How do you get the initial rubric R_0 ? How does it differ from naive?
-
You claim you found errors in the benchmark's evaluator. Can you explain these errors? Perhaps in an Appendix.
-
In section 5, I am completely confused on how many tasks you considered. Initially, you say you produced a rubric for 302, 278 nonvisual. Then you say there are 259 nonvisual. Then you compare values on 242 tasks. Then you say the full task number is 321. Can you please clarify?
-
There are many different "accuracy" metrics and it is unclear to me how they are used. Percentage score is [0,100]. Judge grades are [0,4] then multiplied by 25. Seed grade is [0,1] multiplied by 100. Cell accuracy in [0,1] compares with an expert-written "golden workbook", and an output is considered wrong if this is less than 0.8 (why 0.8?). But on line 287 an output is correct if every cell is correct and 99.8% of preserved cells are unchanged (why 99.8?). And later on line 300 an output is wrong if fewer than 80% match, and a rubric fails an output at below 3/4, why? Can you clarify these choices and definitions?
Suggestion for revision:
- First define the defects generally and then write about how you fix them in your repair loop. Also, define your framework for general benchmarks. Currently, all of this is muddled up together and many things are defined specifically for SpreadsheetBench (for example, cell accuracy).
Response to authors
I appreciate the work the authors have put into grounding this work more solidly. I'm not exactly blown away by this still, but that was my major concern so I'll raise my score accordingly.