acceptodds
Under review as a conference paper at ICLR 2027

When Grading Is Not Mode-Invariant: Answer-Extraction and Normalization Errors in Think/No-Think Comparisons

Abstract

Choosing whether to use an explicit thinking phase requires a reliable estimate of its reported (grader-measured) accuracy gain. Yet on identical MMLU outputs, a vendored Qwen multiple-choice extractor applied to full transcripts reports a think-minus-no-think gap of -15.7 points, while two other mainstream extractors report +17.0 to +17.3; the disagreement persists on the complete MMLU test set. Scoring only the answer segment flips the MMLU gap's sign; one GSM8K extractor's failure persists under the same change. We audit 200,328 Qwen3-8B candidates from five benchmark-derived item sets, using 595 stratified reference labels (LLM-prescreened, human-checked) and separate follow-up and transfer studies. Source-code interventions identify repeated answer-trigger phrases that, in affected outputs, truncate the transcript before final-answer extraction; on 240 Qwen candidates of a separately sampled MMLU follow-up, we evaluate the answer-segment rule whose population verdicts were recorded before that draw. Against the returned question-answer key, the repair removes 47 reference-grading errors and introduces four among the 51 sampled think verdicts that change (51 and zero against benchmark gold). The estimated population error reduction for the 2,560 think outputs has a conditional 95% interval [20.1, 43.9] points; no-think verdicts are unchanged. This does not establish the semantic accuracy ranking. On GSM8K, a last-number extractor's reversal survives the same repair. An exact error decomposition relates mode-specific errors to the reported gap; conditional bounds specify when the evidence supports a comparison; agreement alone cannot certify validity. These results support checking repairs against reference labels and reporting the remaining uncertainty; inference concerns fixed corpora and reference conventions, without adjustment for historical search, undetected label errors, or the post-discovery assembly of the mainstream panel.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM evaluation, answer extraction, measurement error, benchmark methodology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qQQBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

To analyze errors in results measured by evaluators, this study audits 200,328 Qwen3-8B candidates extracted from five benchmark-derived problem sets. The study uses 595 stratified reference labels, as well as individual follow-up and transfer studies. The findings of this study reveal that standard raters exhibit different "think/no-think" reporting patterns for the same output, and that the proposed mode-conditional false positive (FP)/false negative (FN) channel quantifies the underlying errors. Additionally, source code rules and surface rewriting identify deterministic extraction failures, and paired reference audits measure corrected and newly introduced errors. Error decomposition and inversion boundaries pinpoint the potential causes of the observed gaps.

Critical strengths
  • S1: A very detailed analysis has been conducted.

  • S2: This study is being conducted following a review of a vast amount of related research.

  • S3: Theoretical analyses and extensive experiments using multiple datasets have been conducted.

Critical weaknesses
  • W1: This paper is difficult to read and understand. One reason is the large number of parenthetical notes and references to the appendix. Additionally, the experimental results are simply listed without explaining the significance of numerical values.

  • W2: The problem statement should clearly be defined.

  • W3: It is difficult to understand the meaning of the main text without the knowledge provided in the Appendix.

  • W4: Figure 2 should be shown on page 5 or later.

  • W5: All necessary terms should be defined in the text.

Questions for authors

Starting in Section 4, the number of undefined terms increased. Without an explanation of the equations, I could not follow the discussion. Unless the author clarifies the research questions and organizes the text around them, readers unfamiliar with the field will be unable to understand it. Looking at the section on contributions, it appears that the author achieved significant results. Unfortunately, however, I could not grasp the details.

I would appreciate it if you could provide the following three pieces of information:

  • What research questions do you aim to address in this paper?
  • What results did you obtain in response to these questions?
  • Which methods used in your analysis are novel and effective?
1 comment

Thank you for the three direct questions and for noting the detailed analysis, related-work coverage, and multi-dataset experiments. I answer them in order.

1. What research questions do we address, and 2. What results did we obtain?

  • RQ1: Can grading rules reverse a think/no-think comparison on identical outputs? Yes. We held 200,328 Qwen3-8B responses fixed across five benchmark-derived item sets and varied only the grading rules. On MMLU, one extractor reports a think-minus-no-think gap of −15.7 points, while two others report +17.0 and +17.3 on the same outputs. The grading rule therefore changes the reported direction on identical outputs.

  • RQ2: How can we account for grading errors when comparing the two modes? We estimate false-positive and false-negative rates for each mode using stratified reference labels. An exact decomposition links these errors to the observed gap and gives the error-difference threshold at which the reported comparison reverses. At the per-cell level, 5 of 25 task–grader cells support a positive reference-accuracy gap, covering MATH and TriviaQA. After accounting for all comparisons together, the direction remains undecided. The reference-based intervals therefore make clear which comparisons the evidence supports.

  • RQ3: Does removing the reasoning segment reduce grading errors rather than merely raise scores? We count both repaired and newly introduced errors on the same records. In a separate MMLU follow-up of 240 Qwen records, 120 are think records and 51 change verdict. Against benchmark gold, all 51 changes are repairs. The estimated error reduction over all 2,560 think outputs has a 95% interval of [31.4, 44.8] points, while no-think verdicts remain unchanged. On GSM8K, removing the reasoning segment leaves every lm-eval flexible verdict unchanged. The measured repair effect is therefore task-specific.

3. Which methods in our analysis are novel and effective?

Our two contributions are:

  1. Fixed-output evidence and failure mechanisms. We compare identical outputs under different graders, then use source-code interventions to identify the rule responsible for each failure. The paired repair audit applies the same comparison to repaired and newly introduced errors, separating score changes from error changes.
  2. Conditional comparisons. We derive an exact decomposition linking mode-specific false-positive and false-negative rates to the observed gap. The comparison reverses when the sign-aligned error term exceeds the scaled reference gap. Reference-label intervals support a direction when they lie entirely above or below zero. The stratified reference labels provide the mode-specific error rates used in this analysis.

Manuscript revisions. The revised manuscript is organized around these two contributions. It defines the notation in the main text and explains what the numerical results mean. Figure 2 is on page 18 of the revised PDF, satisfying your request that it appear on page 5 or later.

Reviewer_99UoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows when comparing the same model’s performance in thinking mode or non-thinking mode, you can get different results based on how you extract the answer from the model’s generation, that's what they mean by "grading is not mode-invariant".

They audit 200K Qwen3-8B generations on items from 5 benchmarks, with 8 main stream answer grader and 7 in-house grader, and show that those graders can give very different even contradictory comparison results. They argue that for mode evaluation, one should report an observed gain with a reference-based interval, and check repairs against the same labels, counting removed and introduced errors.

Critical strengths

They showed for thinking and non-thinking mode evaluation, standard graders give opposite observed gain direction, showed the importance of answer extraction, and provided insights on how to evaluate LLMs reliably.

Critical weaknesses

(1) This paper is really really hard to read. The authors use many jargons/terms without defining or citing the source. For example, in page 1, line 50, at least you can briefly tell people what “lm-eval” is. In page 4, line 191, please define D_HT and D_res.

I understand that the authors provided a very detailed Appendix, however, I don’t think it’s reasonable to expect the reader to read the 30 page Appendix to understand the paper. At least, define the notations when you use it in the main paper.

(2) Content wise, my main question is about how you get the label S for the 595 records. The process seems messy and lacks detailed description.

For example, in line 214, where is the number “68” from, I understand it’s 32 + 36, but even the 32 and 36 seem random. Since the label S is critical, I think it’s important to show that they are acquired in a clean and trustworthy way.

Questions for authors

I would like to hear more about the process of how you get the labels for the 595 + 400 instances.

1 comment

Thank you for the concrete questions. We first define the notation, then explain exactly how the reference labels were produced.

Notation. lm-eval is EleutherAI's LM Evaluation Harness. DHTD_{\mathrm{HT}} estimates the think-minus-no-think accuracy gap using only the sampled reference labels, with inverse-probability weights to correct for stratified sampling. DresD_{\mathrm{res}} starts with every grader verdict in the fixed corpus and uses the sampled labels to correct grading bias. Thus, DHTD_{\mathrm{HT}} is label-only, while DresD_{\mathrm{res}} combines full-corpus verdicts with a correction.

How the 595 Round-1 labels were produced. We began with 200,328 fixed Qwen3-8B responses across five tasks. We sampled records by task, generation mode, and grader agreement, with more records from disagreement strata. The task counts are 80 GSM8K, 80 MATH, 160 MMLU, 160 MMLU-Pro, and 115 TriviaQA, totaling 595.

For each sampled record, a prescreen LLM saw the question, benchmark gold, and candidate text plus record identifiers. Explicit mode, accuracy, and grader-verdict fields were hidden, although the reasoning text and candidate indices remained visible. The prescreen identified the final committed answer and compared it with benchmark gold under the task's rule. This produced the initial reference label SS: agreement with the stated benchmark convention.

We then added checks rather than replacing those initial labels:

  • 320 multiple-choice records received a mechanical regular-expression re-extraction check.
  • Among the other 275 records, a second LLM reviewed all 32 label–grader mismatch records and 36 stratified concordant spot-check records, giving 68 second-pass semantic reviews. The second LLM worked without access to the first pass's rationales. The 36 concordant records are stratified spot checks.
  • The remaining 207 records use the initial prescreen label SS. Every sampled record therefore has an initial label; 68 also have second-pass semantic review, and 320 have mechanical re-extraction. The full decomposition is 595=68+320+207595=68+320+207.

Two further checks were also within the 595 records. In 132 targeted human confirmations, the reviewer saw the prescreen verdicts and rationales and agreed with all 132. A separate human relabeling of 194 records hid only the prescreen labels and found no disagreement. We report the targeted and random records with their sampling composition. These checks evaluate label reliability alongside the prescreen and second-pass reviews.

The two different 400-record cohorts.

  1. Round-2 semantic extension (400 records): 150 GSM8K no-think, 90 GSM8K think, and 160 MATH think records. This cohort asks whether an answer is semantically correct even when it differs from benchmark gold. A code-oriented LLM prescreened the records. Every record then received either a human confirmation field or a recorded adjudication by the study owner. This process yielded 393 binary judgments and 7 held records. The seven were held because their questions were ambiguous, inconsistent, or under-specified. The Round-1 HT estimate uses the 595 labels; this cohort is used for the semantic-extension analysis.
  2. Answer-extraction follow-up (400 records): a separate later sample for testing answer-segment extraction, including 240 Qwen records. It has its own extraction-repair analysis record.

In the revised manuscript, Section 2 (Study Design) defines DHTD_{\mathrm{HT}} and DresD_{\mathrm{res}}, gives the 68=32+3668=32+36 breakdown, and states explicitly that the 207 records use initial prescreen labels. Section 2 also separates the two 400-record cohorts and explains their different purposes.

Reviewer_s1WkOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I did not understand the AI generated paper. It is very specific too early, sentences do not connect well together -- I actually could not tell what exactly their experimental setup was, what does the results mean.

Critical strengths

None

Critical weaknesses

I cannot assess whether the experiments are sound because I can't understand what each section even means. I could not understand the experimental evidence and how it connects to the main claim.

Questions for authors

No questions, I am unlikely to be convinced to support this paper. Sorry for being overly harsh.

1 comment

Thank you for identifying the three bibliographic errors. We checked each entry against its primary record and confirmed all three:

  1. Lash et al. (2014). Good practices for quantitative bias analysis was published in International Journal of Epidemiology, 43(6):1969–1985 (DOI: 10.1093/ije/dyu149). Our submission listed International Journal of Environmental Research and Public Health, 11(10):10129–10141 instead.
  2. Yang et al. (2024, Qwen2.5-Math). The official record (arXiv:2409.12122v1) lists 16 authors, rather than the 41 authors in our bibliography.
  3. Padmanabhan, Gul, and Goyal (2026, DiSC). The official record (arXiv:2602.16093v1) lists Shankar Padmanabhan, Mustafa Omer Gul, and Tanya Goyal. Our entry omitted Mustafa Omer Gul.

All three works exist, but our bibliographic entries were wrong. We failed to verify them before submission and take full responsibility for this mistake.

We then checked all 52 cited references against primary records, including PubMed, Crossref, DataCite, DBLP, ACL Anthology, arXiv, and official repositories. The check covered existence, author lists, venues, DOIs/URLs, and citation context. It also caught a broken book DOI in Manski (2003), which we corrected from 10.1007/978-0-387-22796-9 to Springer's official 10.1007/b97478, and three GitHub/Hugging Face issue titles that we replaced with the verbatim page titles. These corrections are applied in the revised manuscript.

Your criticism of the presentation is also justified. We held 200,328 Qwen3-8B responses fixed across five benchmark-derived item sets and varied only the grading rules. The revised manuscript explains that setup before the statistical details and organizes the work around two contributions:

  • Fixed-output evidence and failure mechanisms. On identical MMLU outputs, one extractor reports a think-minus-no-think gap of −15.7 points, while two others report +17.0 and +17.3. Source-code analysis shows how repeated answer-trigger phrases can truncate the text and discard the later correct answer. A last-number rule can also select the wrong match even when the final answer is present.
  • Conditional comparisons. We estimate false-positive and false-negative rates for each mode using stratified reference labels. An exact decomposition shows how these errors can change or reverse a comparison. When the reference evidence identifies a direction, we report it with the corresponding interval; otherwise, we report the remaining uncertainty.

We also evaluate parser repairs by counting repaired and newly introduced errors on the same records. In the MMLU follow-up, all 51 changed think verdicts are repairs under benchmark gold. On GSM8K, simply removing the reasoning segment leaves every lm-eval flexible verdict unchanged. The repair analysis belongs to the fixed-output audit and shows why score gains and error reductions need separate measurements.

The revised text defines the notation in the main text and makes these relationships explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.