- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces an approach to codifying natural language rubrics into small non-LLM programs in Python. The authors focus on spreadsheet tasks, and build task-specific verification programs (per individual task rather than class of tasks). The core contribution claimed by the authors is their task-specific codification problem formulation, including failure-based self-improvement loop, and corresponding experimental results demonstrating the effectiveness of their approach. Their validation involves a test set of 36 finance-related spread-sheet tasks, they test their method relative to an LLM-grader's answers on these questions.
Critical strengths
- Detailed and thorough analysis: The authors clearly spent some time carefully analysing the performance of their system within the limited scope of their experiments.
Critical weaknesses
- Small scale and scope of experiments: One type of task (finance-related queries), and only 36 individual tasks (each with 5-6 rubrics). At this scale, we don't have any form of statistical significance to learn reliably from the experiments. To be convincing, this work would need to substantially increase the scale of experiments, both in terms of task types and number of tasks overall. Further the experiments are only relative to LLM-generated labels, no human labels (even if only from authors) are considered, further limiting the amount of information that is provided by the experiments.
- Limited importance of the problem: I believe that making LLM-based graders more efficient for spreadsheet tasks is at best a very niche problem. As LLMs become more efficient, the urgency of this problem further reduces, likely limiting the future impact of work like this. I think there is important eval methodology to be developed, but the direction of this work is not tackling an (in my opinion) important or growing eval problem.
Questions for authors
- Would you be able to compare your approach to https://arxiv.org/abs/2607.22561?
Further suggestions (no need to reply, do not affect score):
- L284: "system SUT" system system-under-test?
- L288: "sys" used but never introduced? SUT?
- L144: What do you mean by strong-model judging, I have not heard that term before
- Fig 2 is way to small, completely unreadable - increase font size please, font in other figures is also unnecessarily small.
Author response for "Official Review of Submission55295 by Reviewer aotV"
We thank the reviewer for their feedback. We address the 2 critical weaknesses and the one question for authors.
We hope the response addresses the reviewer's concerns. We are happy to answer any further questions.