- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper analyzes the effect of introducing specific types of mistakes in an LLM-written proof on LLM judges for those proofs. It finds that LLM judges can respond differently to these introduced mistakes: some models close to ignore harmless ones, while others deduct points for them.
Critical strengths
None
Critical weaknesses
This paper is poorly written, contains only incremental experiments, and has no results of note. I do not understand most of what the authors talk about, except for the main message behind the experiment. Among many other specific flaws, here are some:
- The experiment is performed on a very small scale, with no confidence intervals
- The abstract contains non-sensical or undefined sentences like "We measure that price and propose no grader.", "class is a recorded label (8 of 72 disputed)", "the strict mark", " its held-out gap to the strict mark is exploratory and unresolved", etc. None of these sentences can be understood without reading the entire paper first (and even then they remain somewhat vague).
- There is no sense of story with a motivation, proper setup description, or proper description of the experiments.
- The paper seems mostly LLM-written. Not a problem by itself, but it is clear that the output has not been sufficiently scrutinized for novelty and proper contributions.
Overall, the paper is far away from an interesting contribution.
Questions for authors
See above