acceptodds
Under review as a conference paper at ICLR 2027

Strict Answer Grading can Reverse its Error during Fine-turing

Abstract

Reasoning models are trained, selected and compared by the accuracy of the last boxed answer they write, the default in widely used frameworks. We show that this rule can misreport progress by tens of points, and in opposite directions within one fine-tuning run: early on it credits boxes in reasoning that never finished (at temperature 1 it reports 24.1% on AIME 2025 when only 0.9% of responses deliver a correct answer), and later it misses correct answers stated without a box. Scoring instead the answer each finished response delivers (delivered accuracy) changes four experimental conclusions: a pre-registered fix that raised strict AIME accuracy after an empty thought by 31.0 percentage points did not improve delivered accuracy. On GPQA, strict grading also puts four of twelve public reasoning models at 0.0–11.2% while their delivered accuracy is 35.6–63.6%, and reorders 13 of 66 model pairs. An exact identity, valid for any model, writes the gap as credit for boxes in unfinished reasoning minus correct answers left unboxed, so two counts that need no judge bound it. The bound holds in 237 of 238 cells across three students and twelve public models, the larger count gives the error's sign in 99 of 110 cells where both are nonzero, and removing only the boxes from otherwise identical targets reproduces the late error in a second model (two seeds). Below a combined 2 points, the counts certify a strict score to within about 2 points without a judge; above it, we give a reporting protocol.

Primary area:
datasets and benchmarks
Keywords:
Reasoning models, Evaluation methodology, answer extraction, supervised finetuning, checkpoint selection, distillation, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WRQPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of grading the final boxed answer of a reasoning model, more specifically, grading the answer of the student model when training it using the generations of a teacher model.

They found the strict scoring reads the last boxed answer, which gives credits to boxed answers in generations that are never finished, penalizes answers committed in finished generations but unboxed.

They show that two cheap counts, unfinished responses containing a box and finished responses without a box, bound how wrong a strict score can be. They recommend people to report termination, delivery and correctness separately.

Critical strengths

The problem they study is very practical. Even if they are only studying reasoning models generating answers in boxes, the method can be applied to other output format, such as answers in a fixed JSON format.

I find the two judge-free counts ("a" being the share of responses that are unfinished but contain a box, "b" being the share of responses that finished but contain no box) bounding the error useful, anyone can audit old evaluation logs without running a judge.

Critical weaknesses

First of all, this paper is very hard to read, the language and the writing are hard to follow even for a reader with the right background.

As I mentioned in the strengths, this method can be applied to answers required to be in other fixed format, but I don’t think the authors touch on that.

Questions for authors
  1. Do you know the two teacher model’s performance on the benchmarks you trained the student models on?
  2. Why do you think for most of the student models being trained, the training hurt their performance on the benchmark?
  3. why do you choose step 25 and 200?
2 comments

Thank you for the encouraging review. Each of your questions led to a change in the paper.

Responses to Weaknesses

First of all, this paper is very hard to read, the language and the writing are hard to follow even for a reader with the right background.

We agree, and rewrote the parts a first-time reader meets. The Abstract and a new summary box on p. 2 state the main result in plain terms, and the main text replaces internal labels with direct names (delivered accuracy, prompt mode, grading gap). §3 now has a glossary of the three termination states and two prompt modes. New results sit in two small tables and Fig. 7 (pp. 7, 9–10), with substantive additions in blue; we are glad to clarify any remaining passage.

As I mentioned in the strengths, this method can be applied to answers required to be in other fixed format, but I don’t think the authors touch on that.

Added, following your suggestion: JSON, XML and unit-tested code (§7, Fig. 7b, Appendix T.2–T.3). The two-count bound holds in all 40 JSON/XML settings and all five code settings. The error changes sign under both JSON and XML, though not on every trajectory. Examples from gpt-oss-trained students:

FormatFinding
JSONQwen3-4B on GPQA-main: the error goes from +12.4 pp at step 25 to −9.2 pp at step 200, even with a tolerant JSON parser.
XMLSame model and task: +17.4 pp to −7.0 pp.
Code (LiveCodeBench v6, unit tests, no LLM judge)Qwen3-30B: the official extractor scores 2.3 pp above completed-response scoring at step 25, because it credits code drafts inside unfinished reasoning.

Responses to Questions

Do you know the two teacher model’s performance on the benchmarks you trained the student models on?

Yes; both teachers are now evaluated on all four benchmarks (§3, Appendix T.4). Delivered accuracy (%) at the students' 32,768-token budget:

BenchmarkGemini-3-Flash-Preview (teacher)gpt-oss-120b (teacher)Qwen3-30B before training
AIME 202593.385.081.9
GPQA Diamond90.276.871.7
AIME 202495.885.090.8
GPQA-main82.071.865.8

Teachers are scored on all 30 / 198 AIME 2025 / GPQA Diamond items; Qwen3-30B on 29 / 197, because the first item of each was used for piloting. Gemini was sampled through its API at temperature 0.7; the other two at temperature 1. Both teachers box over 99% of finished benchmark answers here; the Gemini SFT set (0.08% boxed) used a prompt without a box instruction.

Why do you think for most of the student models being trained, the training hurt their performance on the benchmark?

Three candidate factors, measured but not isolated (§3, §6):

  • Outputs overrun the token budget. After 25 steps, 77–96% of AIME 2025 outputs are cut off at the 32k-token limit, against 13–16% before training. At step 200, 24–42% still are, and accuracy is 44.0% (Gemini-trained) and 67.2% (gpt-oss-trained) against 81.9% before training.
  • Imperfect targets. Neither 100k-example training set was filtered for correctness; a verifier flags 20.4% of the Gemini-written solutions as wrong.
  • Little headroom from one teacher. On AIME, gpt-oss-120b (85.0%) is about as strong as Qwen3-30B before training (81.9% and 90.8%).

Training does help in two boxed-target cases (§6, Appendix T.5):

Student, benchmarkDelivered math accuracy: before → afterGrading error on GPQA
Gemma-4-31B-it, standard prompt (one exploratory run, two decoding-seed groups); AIME 2025, step 260070.7% → 75.0%; 60.3% → 73.3%Strict reports a 28.6 / 32.0 pp decline; the delivered decline is 10.8 / 12.3 pp
Qwen3-4B-Base, after empty <think></think>, two training seeds; AIME 2024, step 2005.0% → 19.2% / 18.3%Changes sign in both seeds

why do you choose step 25 and 200?

Step 25 was not hand-picked: checkpoints were saved on a routine 25-step schedule, making step 25 the first saved checkpoint (§3, Appendix A.2). Steps 25–200 formed the pre-registered grid for the initial two-teacher comparison (Appendix O), with step 200 after the early length surge and still inside epoch 1 (no example seen twice). Both sides have been checked:

  • Earlier. A rerun saving every 5 steps shows steps 5–15 behaving like the base model; the length surge arrives abruptly at step 20 (Appendix K).
  • Later. For the Gemini-trained student, strict grading still understates AIME 2024 by 21.7 and 29.2 pp at the last checkpoint, step 885 (in the two prompt modes), and at lower learning rates both phases occur later (§7, Fig. 6a).

On a denser grid for Qwen3-4B (training seed 42, after empty <think></think>), the GPQA-main error shrinks step by step and crosses zero between steps 100 and 150 (Appendix T.5).

AuthorsOpenReview ↗

Brief Summary of Readability Overhaul and New Format Suites

Dear Reviewer WRQP,

As the discussion closes on November 18 (AoE), we wanted to briefly highlight how substantially we revised the paper following your feedback: we rewrote the Abstract, added a plain-language summary box (p. 2) and §3 glossary, and replaced internal labels throughout the main text. We also added the fixed-format suites you suggested across JSON, XML and unit-tested LiveCodeBench v6 code with no LLM judge (§7, Fig. 7b), evaluated both teachers on all four benchmarks (§3), and documented why token-budget overruns and unfiltered targets hurt most runs alongside two boxed-SFT cases that improve, plus a 5-step rerun before our routine 25-step save schedule (§6, Appendix K). We hope the revised manuscript is much easier to read and would be glad to clarify any remaining question before the window closes.

Reviewer_p7gKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the behavior of scoring reasoning models by their last \boxed{} answer. The authors show that during supervised fine-tuning, this rule can overstate accuracy early, by crediting boxes inside reasoning that never finished, and understate it later, by missing correct answers written without a box. They compare it with "delivered accuracy", the answer each finished response delivers. Experiments mainly fine-tune Qwen3-30B-A3B-Thinking on math solutions and evaluate on AIME and GPQA.

Critical strengths

The paper clearly documents that the strict grader can overstate accuracy early in fine-tuning and understate it later, with a large set of experiments.

Critical weaknesses

The contribution is limited.

The core observation is well known: a last-\boxed{} grader can credit a box inside reasoning that never finished, and can miss answers written without a box. The authors' related work cites prior papers showing that answer extraction changes conclusions. The remedy recommended, reading only the answer after the thinking closes and treating unfinished outputs as unanswered, is already an option in NeMo-Skills.

LLM-based answer extractors are also available. In practice, the simple judge-free reader the authors describe already removes most of the checkpoint-selection loss (7.4 points down to 0.6). It is interesting to learn that the error can change sign within one fine-tuning run, but this seems to be too narrow for ICLR.

The evidence for the main claim is narrow, and may not apply to modern models

When the authors add other students, the results are mixed; for example, R1-Distill-Qwen-7B does not show the predicted pattern. Qwen3-30B-A3B-Thinking-2507 before fine-tuning shows no gap, and no frontier close source models are evaluated. This shows strong modern models can potentially handle both format and answer correctly.

Questions for authors

The paper’s largest effects seems to come from an unusual setting, where fine-tuning makes a strong model much worse (Qwen3-30B-A3B-Thinking on Gemini-written solutions: AIME 2025 drops from 81.9% to 44.0%). Does the grading problem still matter in normal training, where fine-tuning improves the model and the training data already use boxed answers?

5 comments

Thank you for the careful review.

Response to Your Question

Does the grading problem still matter in normal training, where fine-tuning improves the model and the training data already use boxed answers?

Yes. In two students where 100%-boxed targets improve mathematics, strict grading still misreports GPQA (§6, p. 7 table, Appendix T.5).

  1. Qwen3-4B-Base, two training seeds. The prompt ends with an empty <think></think>, because the untuned base model never closes its thought. Delivered AIME 2024 accuracy rises from 5.0% to 19.2% / 18.3% by step 200, and the GPQA-main grading error changes sign between steps 25 and 200 in both seeds (+19.8 → −3.8 pp; +20.0 → −2.0 pp).
  2. Gemma-4-31B-it, standard prompting, one exploratory run. Final checkpoint, scored with two groups of decoding seeds:
Decoding groupDelivered AIME 2025: base → step 2600GPQA decline, deliveredGPQA decline, strict
Original70.7% → 75.0%10.8 pp28.6 pp
Fresh60.3% → 73.3%12.3 pp32.0 pp

On AIME, strict and delivered scores agree within 2 pp; on GPQA, strict grading reports the regression at about 2.6 times its delivered size. The audited tools' rule (last box of the whole output, thought included) still overstates the decline, reporting 16.1 / 17.6 pp (supplement). The base score differs between groups (29 AIME items), so the size of the math gain is uncertain; its direction and the GPQA exaggeration hold in both.

Boxed targets do not prevent this, because fine-tuned students still omit the final box on GPQA.

Responses to Weaknesses

The remedy recommended, reading only the answer after the thinking closes and treating unfinished outputs as unanswered, is already an option in NeMo-Skills.

We benchmark exactly that remedy, and it is not enough (§8, Table 3). Row 2 below reads only the answer after the thinking closes and scores every unfinished output as unanswered; it cannot read a completed answer written without a box. Against an LLM judge that reads every completed output, on 314 evaluation settings:

EvaluatorMean selection loss (56 trajectories)Settings off by >2 ppMax errorOutputs judged
Strict last-box (studied rule)7.4 pp194/31460.4 pp0%
</think>-gated last-box (the remedy above)1.3 pp134/31460.4 pp0%
Gated answer-line regex0.6 pp96/31431.9 pp0%
Gated last-token regexabout 047/31412.4 pp0%
Selective judgingabout 00/3141.7 pp15.7%
LLM judge, every completed output (reference)00/314075.8%

LLM-based answer extractors are also available. In practice, the simple judge-free reader the authors describe already removes most of the checkpoint-selection loss (7.4 points down to 0.6).

Agreed for choosing a checkpoint; when the score itself is the result, the regexes we tested are not enough. Your 7.4 → 0.6 is rows 1 and 3; the strongest regex we tested (row 4) reaches about zero. Yet it is still more than 2 pp off in 47 of 314 settings (up to 12.4 pp), which matters when reporting how much a model improved or regressed. Judging every completed output removes this at 75.8% of outputs; judging only those with no parsable box, 15.7%, comes within 1.7 pp.

It is interesting to learn that the error can change sign within one fine-tuning run, but this seems to be too narrow for ICLR.

What matters for training research is that the error moves during training; the sign change is its extreme case. A constant bias cancels when two checkpoints are compared; a moving one does not. Re-read, the same outputs show no gain from a pre-registered intervention (+31.0 pp strict, −3.5 pp delivered) and turn a replicated learning-curve dip into an improvement (−9.1 vs. +31.5 pp); the same error inflates the Gemma regression above.

When the authors add other students, the results are mixed; for example, R1-Distill-Qwen-7B does not show the predicted pattern. Qwen3-30B-A3B-Thinking-2507 before fine-tuning shows no gap, and no frontier close source models are evaluated.

Qwen3-30B shows no gap before fine-tuning and a large one 25 steps later (Fig. 1). Trained on the Gemini recipe, it is then graded 24.1% on AIME 2025 when 0.9% of outputs deliver a correct answer. Six commercial API configurations do have gaps below 1 pp in all 12 settings (Appendix T.4), but four of twelve public open-weight reasoning models are graded 0.0–11.2% on GPQA-main while 35.6–63.6% of their responses deliver a correct answer (§7). Of the added students, R1-Distill-Llama-8B shows both phases for the Gemini recipe; R1-Distill-Qwen-7B does not, because it writes GPQA letters without a box even before fine-tuning (0.6% strict vs. 57.0% delivered).

AuthorsOpenReview ↗

Brief Summary of Revisions and Parser Benchmark Before Discussion Closes

Dear Reviewer SmKi,

As the discussion closes on November 18 (AoE), we wanted to briefly note that the revised paper re-centers its framing (Abstract, p. 2) on why a moving extraction error breaks training comparisons where a constant bias would cancel. New experiments (§§7–8, Table 3) show termination gating and our strongest gated regex still erring by >2 pp in 134/314 and 47/314 settings (vs. 0/314 when judging the 15.7% completed unboxed outputs), and show tolerant json_repair plus gating enlarging step-200 GPQA-main error (−9.2 → −16.0 pp) when completed answers shift into prose. Please let us know if our response and revision address your concerns, or if we can clarify anything further before the window closes.

The new commercial-API results have a much lower gap, which supports my concern. The larger gaps reported in this paper often come from early 2025 distilled models, such as R1-distill-Qwen-7B. The problem that this paper is trying to address is probably no longer critical for most of the modern models. Given this, I will keep my score.

Reviewer_SmKiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the biases introduced during model fine-tuning by evaluation rules that strictly rely on boxed answers: such rules may prematurely credit correct answers derived from incomplete reasoning, while conversely failing to recognize correct answers that are not presented in the required boxed format. Through an analysis of training trajectories, controlled experiments on output formatting, and evaluations of publicly available models, the authors demonstrate how these biases influence learning curves, model rankings, and the selection of model checkpoints. Finally, they propose an evaluation approach that incorporates both the model's termination state and more flexible methods for answer extraction.

Critical strengths
  1. The figures and tables are very well-executed, and the presentation has been carefully polished. The writing is fluid, logically sound, and highly readable.
  2. The experiments are very thorough, accounting for a wide range of scenarios; this reduces the likelihood that the key findings are merely an artifact of a specific model or scoring method.
Critical weaknesses
  1. The motivation for this paper leans too heavily toward engineering-related issues and is quite weak. It rests on the assumption that those training the models fail to employ better, non-naive parsers. This is akin to analyzing why a missing "}" in JSON causes a code error—a problem that could easily be avoided by simply following proper parsing rules.

  2. While the paper provides an exhaustive empirical analysis of potential flaws in answer extraction, its primary new insights are limited to the magnitude of the resulting bias and how it evolves during model training. It is common knowledge that format-based extraction can introduce bias into benchmarks; this is hardly a novel discovery. Although the quantitative experiments are commendable, the paper fails to demonstrate how these findings constitute a conceptual contribution that significantly advances the existing understanding of benchmark evaluation.

  3. The paper’s own results indicate that implementing simple termination handling and robust answer extraction eliminates most of the practical consequences. Consequently, it remains unclear what additional value the proposed comprehensive diagnostic process offers compared to simply fixing the evaluators directly.

Questions for authors

See weaknesses

2 comments

Thank you for the generous assessment of the experiments and presentation.

Responses to Weaknesses

While the paper provides an exhaustive empirical analysis of potential flaws in answer extraction, its primary new insights are limited to the magnitude of the resulting bias and how it evolves during model training.

Agreed: extraction bias itself is known. What is new, as you note, is how it evolves during training, because standard controls of training experiments miss a bias that moves with the model (§§4–6). Benchmark studies treat extraction bias as fixed for a model and parser. During fine-tuning the model keeps changing, and the parser's error moves with it: early it credits boxes inside unfinished reasoning, later it misses completed unboxed answers. Identical evaluation code across checkpoints therefore does not make scores comparable. Two results that had passed the usual safeguards change when the same outputs are re-read (Gemini-trained Qwen3-30B, AIME 2025):

Training comparisonStrict changeDelivered change
Pre-registered retrain adding gold boxes to targets (after empty <think></think>)+31.0 pp−3.5 pp
Learning curve, steps 25 → 75 (standard prompt; mean of two fresh training seeds)−9.1 pp+31.5 pp

The retrain passed its pre-registered test with no delivered gain, and the dip replicated on fresh seeds while delivered accuracy rose: pre-registration and replication guard against noise and analyst choices, not a moving measurement error. The paper also decomposes strict − delivered into credit for unfinished outputs − missed completed answers plus parser–judge disagreement on parsed completions, with the first two terms bounded directly from existing logs.

The paper’s own results indicate that implementing simple termination handling and robust answer extraction eliminates most of the practical consequences. Consequently, it remains unclear what additional value the proposed comprehensive diagnostic process offers compared to simply fixing the evaluators directly.

Fixing the evaluator requires knowing that the fix worked; partial fixes leave error and can enlarge it (§8, Table 3). Against an LLM judge that reads every completed output, on 314 evaluation settings:

EvaluatorSettings off by >2 ppMax errorOutputs judged
Termination gating134/31460.4 pp0%
Gating + strongest regex tested (last-token)47/31412.4 pp0%
Selective judging0/3141.7 pp15.7%
LLM judge, every completed output (reference)0/314075.8%

The strongest regex we tested is as good as selective judging for choosing a checkpoint on average. When the score itself is the result, it is still off in 47 settings; a broader parser reads more answers but can also misread more. The diagnostic adds three things. From existing logs, it shows where a gated parser can be trusted: settings where at most 2% of outputs are completed without a parsable box can skip the judge (117/314), adding at most 2 pp beyond parser–judge disagreement. It sends only unparsed completed outputs to the judge, 15.7% overall. And it warns that the two errors offset each other, so fixing one alone can enlarge the total (JSON test below).

It rests on the assumption that those training the models fail to employ better, non-naive parsers. This is akin to analyzing why a missing "}" in JSON causes a code error—a problem that could easily be avoided by simply following proper parsing rules.

The naive rule is the shipped default, and a tolerant parser does not remove the error (§7, Appendices H.7, T.2). The NeMo-Skills, NeMo-RL and verl implementations we audited read the last box of the whole output by default, and none separates truncated from finished outputs. Under our studied last-box rule, four of twelve public reasoning models score 0.0–11.2% on GPQA-main while 35.6–63.6% of their responses deliver a correct answer. We also tested the JSON analogy directly: we requested JSON answers and scored them with a tolerant parser (json_repair), then added termination gating. Error against delivered accuracy for gpt-oss-trained Qwen3-4B on GPQA-main (gated row: supplement):

JSON scoringStep 25Step 200
json_repair+12.4 pp−9.2 pp
json_repair + termination gating0.0 pp−16.0 pp

A missing brace is a syntax error that a parser can repair. Here there is nothing to repair: by step 200 the model states its final answer in prose, which no JSON parser reads. Gating even enlarges the step-200 error (9.8% scored vs. 25.8% delivered), because credit for drafts in unfinished outputs had been masking part of the missed answers. If schema compliance is the requirement, that is a legitimate failure; as a measure of answer correctness, it is off by 16 pp.

AuthorsOpenReview ↗

Brief Summary of New Experiments Before Discussion Closes

Dear Reviewer p7gK,

As the discussion closes on November 18 (AoE), we wanted to briefly highlight our new experiments on 100%-boxed fine-tuning where math improves (§6, p. 7): GPQA error still flips sign across two Qwen3-4B-Base seeds (empty-thought prompt) and exaggerates the GPQA decline in an exploratory Gemma-4-31B-it run across both decoding groups (28.6 / 32.0 pp strict and 16.1 / 17.6 pp whole-output, vs. 10.8 / 12.3 pp delivered). Benchmarking the exact NeMo-Skills </think>-gating option and stronger regexes across 314 settings (§8, Table 3, plus six commercial API controls at <1 pp in §7) shows gating and last-token regex leaving 134/314 and 47/314 settings >2 pp off (despite matching average checkpoint selection), whereas judging only the 15.7% completed unboxed outputs leaves 0/314. Please let us know if our response and revision resolve your concerns, or if anything remains that we can clarify before the deadline.

Comments on the paper

3 comments
AuthorsOpenReview ↗

GLobal Response

Thank you for the constructive reviews. We are glad the reviewers found the experiments "very thorough" (SmKi), the effect clearly documented (p7gK), and the two judge-free counts useful for auditing existing logs (WRQP).

Our central contribution is a controlled diagnosis of why fixed answer graders mismeasure fine-tuning progress, with zero-judge log checks where evaluation breaks. In our main runs, last-box grading first credits boxes inside unfinished reasoning and later misses completed answers written without a box. Because the error moves and even changes sign across checkpoints, holding evaluation code fixed does not hold measurement error fixed, and standard experimental safeguards miss it:

Training comparison (Gemini-trained Qwen3-30B, AIME 2025, §5)Strict changeDelivered change
Pre-registered retrain adding gold boxes to targets (empty-thought prompt, step 200)+31.0 pp−3.5 pp
Learning curve, steps 25 → 75 (standard prompt; mean of two fresh training seeds)−9.1 pp+31.5 pp

Here strict grading (our studied rule) reads the last box after </think>, or anywhere in the output if </think> never appears; the NeMo-Skills, NeMo-RL and verl implementations we audited read the last box of the whole output by default. Delivered accuracy scores the final visible answer of completed outputs (via a gold-blind LLM judge) and gives unfinished outputs zero. An empty-thought prompt ends with an empty <think></think>.

New experiments address the main concerns:

Reviewer questionEvidence and finding
Does it persist when 100%-boxed SFT helps? (p7gK, WRQP)Prompted after an empty <think></think>, Qwen3-4B-Base rises from 5.0% to 19.2% / 18.3% on AIME 2024 across two training seeds while GPQA-main error flips sign in both (+19.8 → −3.8; +20.0 → −2.0 pp). In one exploratory Gemma-4-31B-it run (standard prompt, step 2600), AIME 2025 rises in both decoding-seed groups (70.7% → 75.0%; 60.3% → 73.3%) while GPQA falls 10.8 / 12.3 pp delivered versus 28.6 / 32.0 pp strict (16.1 / 17.6 pp under the toolkits' whole-output rule; §6, Appendix T.5).
Do termination gating and stronger regexes suffice? (SmKi, p7gK)Across 314 archived settings (100,504 outputs), </think> gating leaves 134/314 settings >2 pp from a full completed-output judge (max 60.4 pp). Gated last-token regex matches selective judging for average checkpoint picking, yet leaves 47/314 >2 pp off (max 12.4 pp). Selective judging, which sends only completed outputs without a parsable box to the judge (15.7% vs. 75.8% of outputs), leaves 0/314 >2 pp off (max 1.7 pp), and stays within 2 pp on 33 of 36 non-overlapping boxed-SFT settings (§8, Table 3, Appendix T.1).
Does it transfer beyond boxes, or without an LLM judge? (SmKi, WRQP, p7gK)On 4B GPQA-main, tolerant json_repair flips +12.4 → −9.2 pp, and adding gating enlarges the step-200 error to −16.0 pp as completed answers appear in prose. On LiveCodeBench v6 (unit tests, no LLM judge), official extraction credits <think> code drafts (+2.3 pp at step 25). Six commercial API setups stay <1 pp across all 12 settings (§7, Appendix T.2–T.4).

What the revised paper claims: Termination gating, regex extractors and LLM judges are existing tools; our addition is showing that fine-tuning moves their error across checkpoints, decomposing that error and bounding its two main terms from existing logs (up to parser–judge disagreement), and mapping where cheap parsers suffice (average checkpoint picking; <1 pp commercial APIs) versus fail (47/314 settings). Sign reversal is not universal (e.g., R1-Distill-Qwen-7B starts unboxed on GPQA).

Manuscript revisions (substantive additions in blue): (1) rewrote the Abstract and p. 2 summary box around training conclusions; (2) added p. 7 Gemma endpoint and denser 4B trajectories (§6, App. T.5); (3) added the 314-setting parser benchmark (§8, Table 3, Fig. 7c, App. T.1); (4) added JSON/XML, zero-judge code, commercial APIs and teacher baselines (§3, §7, Fig. 7b, App. T.2–T.4); (5) added a §3 state/prompt glossary, replaced internal terms in main prose (delivered accuracy, prompt mode, grading gap), and fixed the title typo (Fine-Tuning).

AuthorsOpenReview ↗

Final Hours of Author-Reviewer Discussion: Authors Available for Any Questions

Dear Reviewers and Area Chair,

With only a few hours left before the Author-Reviewer Discussion window closes (November 18, 23:59 AoE), we wanted to briefly note that we remain online and can respond promptly to any last-minute questions while author replies are still open. Our revised manuscript and point-by-point responses include substantial new experiments and readability rewrites directly addressing each review, and we would be very grateful to know if anything remains unclear. Thank you again for your time and thoughtful feedback.

AuthorsOpenReview ↗

Summary for the Area Chair: contribution, discussion status, remaining disagreement

Dear Area Chair,

In brief. Both negative reviews share a central objection: the bias is known, and simple fixes remove most of its practical consequences. The revision tests those fixes. Even the strongest extraction rule we tested leaves 47 of 314 accuracy estimates more than 2 percentage points from the LLM reference. Gemma further shows that a small grading error before fine-tuning can become a large one during training.

The contribution in three points

  1. A fixed grader does not guarantee a fixed error. It can first credit answers inside unfinished reasoning, then miss correct completed answers without a box. These errors change during training and distort comparisons between checkpoints.

  2. This changes experimental conclusions. A pre-registered intervention raises strict accuracy by 31.0 percentage points, while accuracy on the answers actually delivered changes by −3.5 points. In a separate experiment repeated with two training seeds, a 9.1-point strict decline accompanies a 31.5-point improvement in delivered-answer accuracy.

  3. The error can be audited and reduced. Two counts from existing logs expose these error sources. Our correction judges 15.7% of all outputs instead of 75.8%, and stays within 2 percentage points of the full LLM reference in all 314 archived settings.

The reviewers describe the experiments as “very thorough” (SmKi), the sign change as “interesting” (p7gK), and the problem as “very practical” (WRQP).

What the reviewers asked for, and what we added

• Existing fixes benchmarked (SmKi, p7gK): direct comparisons in §8, Table 3. • Useful training with boxed targets (p7gK): two students in §6; the Gemma result is one exploratory training run. • Frontier closed-source models (p7gK): six API configurations, with gaps below 1 percentage point, reported as a limit on applicability (§7). • Teacher baselines, other output formats, readability (WRQP): added in §3, §7 and Appendix T.

Why the main objections do not settle the case

“simple termination handling and robust answer extraction eliminates most of the practical consequences” (SmKi; p7gK raises the same point). The strongest extraction rule matches our method for checkpoint selection on average. Yet substantial errors remain in individual accuracy estimates. Across the same 314 settings, Table 3 counts estimates more than 2 percentage points from the LLM reference:

• Strict grading after checking that reasoning has ended: 134 settings. • The strongest answer-extraction rule we tested: 47 settings. • Our correction: 0 settings.

Good average checkpoint selection does not ensure an accurate estimate of how much a training intervention helped or hurt.

“It rests on the assumption that those training the models fail to employ better, non-naive parsers” (SmKi). The NeMo-Skills, NeMo-RL and verl implementations we audited use last-box rules by default (Appendix H.7).

“probably no longer critical for most of the modern models” (p7gK’s stated reason for keeping the score). Small commercial-API gaps do not test whether grading stays reliable during fine-tuning. In Gemma-4-31B-it under standard prompting, the GPQA grading error grows from 4.2 percentage points before training to 38.1 at step 1000, within the same decoding group (Table 55). The training targets were boxed, and mathematics improved at the final checkpoint. A small initial gap does not rule out a large error during training.

The judgment we ask the AC to make

A constant bias would cancel between checkpoints. This one changes during training and reverses experimental conclusions. We ask the AC to assess the contribution demonstrated here: exposing that failure in a pre-registered intervention and a replicated learning curve, testing the proposed fixes, and providing a practical audit and correction.

Limits we retain

The Gemma result is a single exploratory run. The strongest extraction rule matches our method’s average checkpoint-selection performance. The scoring comparisons and formal guarantees are relative to an LLM reference judge, not human labels.

This note is visible to everyone. If we have misstated a reviewer’s position, we welcome a correction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.