acceptodds
Under review as a conference paper at ICLR 2027

Little Engines of Evidence: Codification of Task-Specific Natural Language Rubrics

Abstract

Language-model-based evaluators can grade outputs of complex tasks using natural-language rubrics as specifications. High-quality evaluators often use agentic workflows that make multiple LLM and tool calls. While effective, such evaluators are expensive, noisy, and difficult to run at high cadence. We present an approach for rubric codification: converting each natural-language rubric into a small executable program that servers as the evaluator. We focus on spreadsheet tasks where an input workbook is transformed into an output workbook based on a user query, and codify the NL rubrics associated with such tasks. A codification agent receives a rubric, training examples consisting of initial and final workbooks plus verdicts from an (expensive) agentic evaluator, and it emits Python code. We evaluate the effectiveness of the codification approach on a finance-spreadsheet benchmark set with 36 queries and 194 rubric criteria.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Agents, Evaluation, Multi-Agent Systems, Program Synthesis, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aotVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces an approach to codifying natural language rubrics into small non-LLM programs in Python. The authors focus on spreadsheet tasks, and build task-specific verification programs (per individual task rather than class of tasks). The core contribution claimed by the authors is their task-specific codification problem formulation, including failure-based self-improvement loop, and corresponding experimental results demonstrating the effectiveness of their approach. Their validation involves a test set of 36 finance-related spread-sheet tasks, they test their method relative to an LLM-grader's answers on these questions.

Critical strengths
  1. Detailed and thorough analysis: The authors clearly spent some time carefully analysing the performance of their system within the limited scope of their experiments.
Critical weaknesses
  1. Small scale and scope of experiments: One type of task (finance-related queries), and only 36 individual tasks (each with 5-6 rubrics). At this scale, we don't have any form of statistical significance to learn reliably from the experiments. To be convincing, this work would need to substantially increase the scale of experiments, both in terms of task types and number of tasks overall. Further the experiments are only relative to LLM-generated labels, no human labels (even if only from authors) are considered, further limiting the amount of information that is provided by the experiments.
  2. Limited importance of the problem: I believe that making LLM-based graders more efficient for spreadsheet tasks is at best a very niche problem. As LLMs become more efficient, the urgency of this problem further reduces, likely limiting the future impact of work like this. I think there is important eval methodology to be developed, but the direction of this work is not tackling an (in my opinion) important or growing eval problem.
Questions for authors
  1. Would you be able to compare your approach to https://arxiv.org/abs/2607.22561?

Further suggestions (no need to reply, do not affect score):

  1. L284: "system SUT" system system-under-test?
  2. L288: "sys" used but never introduced? SUT?
  3. L144: What do you mean by strong-model judging, I have not heard that term before
  4. Fig 2 is way to small, completely unreadable - increase font size please, font in other figures is also unnecessarily small.
1 comment
AuthorsOpenReview ↗

Author response for "Official Review of Submission55295 by Reviewer aotV"

We thank the reviewer for their feedback. We address the 2 critical weaknesses and the one question for authors.

  1. Regarding small scale and scope of experiments, we have extended our codification evaluation to a larger dataset with 243 tasks and approximately 3000 rubrics that cover both correctness and presentation quality. The main finding was that the numbers do not change, with agreement remaining around the reported 80% number. Additionally, we also codified the rubrics of the public GDPval dataset, which we intend to release publicly. We are getting similar agreement numbers there. We plan to add these results to a revised version of the paper.
  2. Regarding the niche problem comment, the GDPval codification demonstrates that the codification approach extends to a more general class of artifacts. Further, even as efficiency increases, run-to-run variance as well as interpretability remain problems that programmatic graders address. As things stand currently, we are able to eliminate the variable cost of long evaluation runs with a fixed one-time cost of codification, as well as improve latency 70%. We will revise the paper to emphasize the importance of codified rubrics and arguing why it is not a niche thing.
  3. The provided paper is a good validation for the importance of codified rubrics. A key difference is that they consider "task-agnostic rubrics" (or rubrics that apply to all tasks). We, on the other hand, consider codification of task-specific rubrics -- rubrics that are specific to this one task. The task-specificity is crucial for why codification using "by-example program synthesis" works, as we argue in the paper. In fact, "task-agnostic rubrics" are harder to codify -- the cited paper has to use "committee of expert programs" and fallback mechanisms for that reason. Task-specific rubric codification is a more promising direction and equally impactful since several datasets are now coming with task-specific rubrics. Task-specific rubric codification also allows us to quantify the extent to which a certain rubric is better than the other. Further, the cited paper shows parity with performance on models upto 13B parameters, while our method shows parity with a frontier model as it’s judge. So, overall, the cited work is very closely related to our paper, but we make some important points that are significantly different that may help with clarifying the potential of rubric codification and its impact.

We hope the response addresses the reviewer's concerns. We are happy to answer any further questions.

Reviewer_6XAeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes rubric codification: converting task-specific natural-language rubrics into small executable Python programs that approximate an expensive agentic LLM grader. The problem is framed as programming-by-example. A codification agent is given the rubric, input/output workbooks, and the grader’s scores and explanations, then writes evaluator code, refining it with a CEGIS-style feedback loop and falling back to the LLM grader when the code abstains. Experiments on an in-house finance-spreadsheet benchmark (36 queries, 194 rubrics) report that the codified evaluators largely agree with the agentic grader.

Critical strengths
  1. The problem is practical: agentic graders are costly and noisy, which makes frequent regression testing expensive.
  2. The PBE framing is clear, and the held-out runs are collected more than a month after codification, a realistic setting in which the system under test keeps evolving.
  3. Some observations are interesting: disagreements concentrate on instances where the grader itself is unstable, errors within a rubric tend to be one-sided, and grader explanations are important supervision for code synthesis.
Critical weaknesses
  1. The scope is very narrow and the method is not general. It covers one domain, one in-house system under test and one grader. The synthesized code relies on hard-coded strings and vocabularies drawn from the sampled outputs, so it is unlikely to transfer to other systems, even though comparing systems is the main purpose of a benchmark.
  2. The evaluation metric is not informative under the class imbalance. Figure 2 shows that the grader passes about 81% of instances, so an always-pass baseline already reaches roughly the best reported agreement (0.817), and every other variant in Tables 1–2 falls below it. The paper does not report a chance-corrected or balanced metric.
  3. No baselines are included. Natural comparisons would be a cheaper single-call LLM judge, a checklist-based lightweight judge such as RocketEval, and the grader's own run-to-run agreement. There is also no human-labeled ground truth, and the claimed cost savings are estimated rather than measured.
  4. Technical novelty is limited, since LLM program synthesis with execution feedback and synthesized verifier programs are well established. The benchmark, system under test and grader appear to be private, which limits reproducibility.
Questions for authors

Please check my concerns in weaknesses.

1 comment
AuthorsOpenReview ↗

Author response for "Official Review of Submission55295 by Reviewer 6XAe"

We thank the review for their feedback. We address the 4 points in the critical weakness below:

  1. To address the concern about scope being narrow, we have now successfully generated code graders for the tasks on the open-source GDPval dataset, which we intend to release publicly. Since our approach for codification requires sample solutions and their grades, we generated sample solutions using 9 different models from different size classes across 4 model families. Our results hold on GDPval codification too, with the code-graders and grading agent agreeing around 80% of the time. Regarding the presence of hard-coded strings in the generated code, we note that it is one of the main points of the paper-- the hard-coded strings work because the code is expected to grade the output of an agent for a fixed task and fixed rubric. Queries usually specify a sheet name as shown in the example, in which case looking at hardcoded strings makes sense. When the query is open-ended, the input is still fixed, and the diversity of outputs presented to the codification agent ensure that the agent generates code that generalizes the constants that are not directly fixed by the queries. (And LLMs are good at performing this generalization.)
  2. Regarding the concern of class imbalance and reporting chance-corrected metrics, note that we provide counts for all classes, namely TT, TF, FT, FF, where the first T/F denotes the result of the agentic grader and the second T/F denotes the verdict from the codified grader. We can easily use these numbers to report the requested metrics.
  3. Regarding other baselines, note that lightweight LLM-judges are not interesting here because we are specifically trying to get a replacement of expensive LLM-as-a-judge with codified judges. The main point is here is showing the feasibility of using codified rubrics.
  4. To address the concern around human evaluation, we looked at a subsample of 150 disagreements. We found that the codified judge was correct (agreed with human) about 60% of the time (only looking at disagreements), whereas the agentic judge was “correct” (agreed with human) the remaining 40% of the time. However, there is an important caveat here: in most of the disagreement cases, there is an element of human subjectivity involved since pass/fail is not entirely determined by the rubric (and solution). For example, one rubric read: “Layout-Cell sizing Rows and columns in added dashboard areas should be sized so labels and values are readable without excessive wrapping, truncation, or wasted space.” The solution has 3 tables out of 10 containing some truncated text and one grader scores it a 3/4 (pass) while the other scores it a 2/4 (fail). The main point though is that the codified grader is no worse in any aspect than an LLM-based grader, but it is beneficial in terms of repeatability, latency, and cost.

We sincerely hope that the answers here address some of the critical concerns of the reviewer. We are happy to answer additional questions. We truly believe in the value of codifying rubrics for all public datasets and using codified checks as a means of performing repeated evaluations.

Reviewer_NroQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies task-specific rubric codification: synthesizing reusable Python graders from natural-language criteria, example workbooks, and an LLM grader's scores and explanations. The approach uses counterexample-guided refinement and falls back to the LLM grader when codification fails or the program abstains. Experiments cover 36 finance spreadsheet tasks and 194 rubric criteria, with ablations on examples, explanations, and CEGIS.

Critical strengths
  1. The task-specific formulation addresses a practical bottleneck in repeated agent evaluation and produces reusable, inspectable grading programs.

  2. The ablations demonstrate the value of grader explanations and execution feedback. CEGIS improves pass/fail agreement from 69.4% to 81.7%, and testing on outputs collected over a month later provides some evidence of robustness to system changes.

Critical weaknesses
  1. Limited novelty and contribution. The main contribution is to turn an LLM grader's judgments and explanations into executable code for fixed tasks using existing methods for generating code and improving it through testing and feedback. I find this combination incremental, with limited methodological novelty.

  2. Limited evaluation. Experiments cover only one finance spreadsheet benchmark with 36 tasks and use GPT-5 reasoning for grading and code synthesis. The narrow task coverage, absence of cross-model validation, and lack of direct comparisons with prior methods limit the evidence for the approach's broader utility.

  3. Insufficient validation of grading disagreements. Figure 2 groups cases with inconsistent LLM verdicts as Open and suggests that the code verdict can be considered correct. However, reference-grader inconsistency does not establish code correctness; without independent adjudication, it remains unclear whether these disagreements reflect code errors, grader errors, or ambiguous rubrics.

Questions for authors
  1. What is the main methodological contribution beyond combining task-specific code generation with example supervision and CEGIS?

  2. Can the authors provide comparisons with direct rubric-to-code synthesis under a matched budget, or results on another benchmark or model?

  3. Can the authors independently adjudicate a sample of code/LLM disagreements, including Open cases, and report which evaluator is correct?

2 comments
AuthorsOpenReview ↗

Author Response to Official Review of Submission55295 by Reviewer NroQ

We thank the reviewer for their feedback. We answer the three questions below.

  1. The main contribution is the idea that for task-specific rubrics, LLM-based grading can be replaced by code-based grading, and this can be achieved using “by-example program synthesis (that includes CEGIS)”. Note that we are not “combining task-specific code generation with example supervision and CEGIS”, we are casting the “task-specific rubric codification” problem as an “by-example program synthesis” problem. The key contribution is the realization that the input and output space of the (expected) code is sufficiently constrained to make the approach feasible. We believe that this is a subtle, but very important point, with huge potential impact (and we are seeing it play out too).
  2. Regarding limited evaluation, we have extended our codification evaluation to a larger dataset with 243 tasks and approximately 3000 rubrics that cover both correctness and presentation quality. The main finding was that the numbers do not change, with agreement remaining around the reported 80% number. Additionally, we also codified the rubrics of public GDPval dataset, which we intend to open-source, and where we see similar numbers. We plan to add these results to a revised version of the paper.
  3. To address the concern around human validation, we looked at a subsample of 150 disagreements. We found that the codified judge was correct (agreed with human) about 60% of the time (only looking at disagreements here), whereas the agentic judge was “correct” (agreed with human) the remaining 40% of the time. However, there is an important caveat here: in most of the disagreement cases, there is an element of human subjectivity involved since pass/fail is not entirely determined by the rubric (and solution). For example, one rubric read “Layout-Cell sizing Rows and columns in added dashboard areas should be sized so labels and values are readable without excessive wrapping, truncation, or wasted space.” The solution has 3 tables out of 10 containing some truncated text and one grader scores it a 3/4 (pass) while the other scores it a 2/4 (fail). The main point though is that the codified grader is no worse in any aspect than an LLM-based grader, but it is beneficial in terms of repeatability, latency, and cost. We sincerely hope that the reviewer finds the response helpful. We are happy to answer any further questions.
Reviewer_NroQOpenReview ↗

Response after rebuttal

Thank you for the response, but my main concern has not been sufficiently addressed, so I will maintain my original rating of 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.