- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors asked five models the same math and multiple-choice questions but changed how the final answer had to be written. Whether answers were correct did not changed model rankings, but whether answers were correct AND correctly formatted reversed rankings 12 times. The authors suggest that benchmarks should report correctness and format compliance separately to disentangle this observation .
Critical strengths
This conducts careful and through investigation with paired comparisons, multiplicity correction, frozen scorers, no manual overrides, and full release of the raw outputs. The findings to report correctness, compliance, and strict accuracy separately are practical, sensible and easy to follow through.
The authors also share results that go against their proposed idea honestly which helps readers make more informed decisions and interpretation, e.g. Qwen32’s sharp drop in performance under the revised prompt and the incomplete MATH confirmation
Critical weaknesses
The paper appears more like a rigorous technical report, where the novelty and scope of the investigation is a bit thin for a main track paper. On this front, a stronger argumentation and a richer set of analyses and discussions could help strengthen the intellectual framing to inspire deeper resarch questions and discussions.
This paper is also farily dense and lacks narrative writing to make it more human readable.
Questions for authors
How much of the strict-reversal count remains if you only use the bracket-free prompt from the start? Was there analysis to show why or how models differ in compliance?
How model-dependent are the findings? Do the reversals appear with current frontier or reasoning models? From your observations, which model properties predict format compliance?
Thank you for your response.