- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Sometimes answers are marked incorrectly for math benchmarks because of formatting issues.
Critical strengths
It's obviously true.
Critical weaknesses
It's not obviously important.
Questions for authors
[I did this first pass with my own faulty eyes and brain, i haven't yet used AI to go through it, I want to have a discussion first]
-
Could you elaborate a bit on why the reader should care? the methodology and whatever are obviously fine, but even if you did this for 100 models, I struggle to understand why this is important or surprising.
-
I don't see a lot of discussion or analysis as to why this is happening (merely that it is happening), nor do any constructive insights pop out at me, perhaps you can tell me what you think the takeaways are.
-
Please answer honestly and without AI; did you really need 12 pages to get the message across? Either I am missing something incredibly important somewhere in the article, or this really is a simple factual observation measured for two open source models inflated to be the length of a paper. Please educate me.
2 comments
yea, no.
history will look poorly upon this era when we flooded the epistemic commons with such pollution.
Thank you for asking directly. The reason to care is that extraction does not merely move an accuracy number slightly; it can change the scientific conclusion drawn from the same model outputs. Under strict scoring, Qwen and Llama score 45.2% and 0.4%, a gap of 44.8 percentage points. Reading terminal answers from the identical responses gives 47.7% and 28.1%, reducing the apparent gap to 19.6 points without changing either model. The estimated cue effect also changes: for Qwen it moves from +1.25 points under strict scoring to -0.4 points under terminal scoring, while for Llama it moves from -0.83 to -2.9 points. A model comparison or prompting conclusion can therefore be partly a property of the evaluator rather than the generated reasoning.
We also checked that this is not peculiar to our two extractors. Math-Verify 0.8.0 extracts from 42.7% of the 960 frozen responses under an anchored configuration and from 100% under its default configuration. For the 160 percentage-family responses, raw-number and percentage-aware comparison changes its correct count from 1 to 103. The concrete mechanism is therefore not one isolated parser bug: answer anchoring, format compliance, unit interpretation, and truncation each determine what enters the final score.
The constructive takeaway is an evaluation procedure. Before generation, specify the response contract, extraction anchor, and unit semantics. After generation, report extraction availability, correctness conditional on extraction, and full-denominator accuracy separately. Finally, rerun the conclusion with an established alternative extractor on the same frozen outputs. This makes it possible to tell whether a claimed model or prompting improvement survives the measurement choice.
The central message can indeed be stated briefly. The value of the full study is the controlled evidence behind it: 240 programmatically checked problems, paired prompt interventions, 960 released responses, family-level uncertainty, token-cap diagnostics, and an independent Math-Verify audit. Together these show not only that extraction matters, but which conclusions change, why they change, and how an evaluator can detect the problem before reporting a benchmark result.