Free-response benchmarks do not score model text directly: an answer extractor first decides which number counts as the answer, so two pipelines scoring the same outputs can report different gaps. We make that scoring step the object of study. For a frozen class of seven deterministic extraction rules, we compute every gap fixed outputs can yield when one shared rule scores every response, and how far the range expands when the rule may vary per question or per system—oracle stress tests of assignment consistency. A registered reanalysis of a disclosed development cohort (22 comparisons, four 1–2B families) and a preregistered held-out confirmation locate both robustness and fragility, and the pipeline falsifies its own development sign flip. On 193 harder ContextMATH pairs the audit separates two regimes: Qwen2.5's scenario effect has no unique value until the extraction rule is declared (a shared-rule range of −13.0 to +12.4 points at 7B, −14.5 to +8.8 at 32B—rule dependence, not a confidence interval), while four newer checkpoints from two families earn exact within-class robustness certificates: one identical negative gap under every rule and assignment. Bigger models alone do not remove scoring ambiguity. Declare one extractor before scoring, apply it to every system, and report the shared-rule range over a prespecified class.
Thank you for the review. Your question deserves a direct answer.
The point of the paper. Benchmark numbers for free-response tasks depend on an answer extractor that is almost never declared. Holding model outputs fixed, we compute every gap a prespecified class of extractors can produce under several assignment disciplines. When the gap is the same under every rule, the comparison earns a certificate; when it is not, the paper reports the range instead of a sign. Concretely: the same saved Qwen2.5-7B outputs on a hard benchmark yield a scenario effect anywhere from −13.0 to +12.4 points depending on which deterministic extractor scores them (we now also show which convention is responsible), while four newer checkpoints from two families yield one identical negative gap under every rule.
On the ethics flag. The paper's mandatory AI use statement discloses that generative AI assisted with study design, formal reasoning, implementation, analysis, literature review, and writing, and that the author reviewed all of it and takes full responsibility, as the ICLR 2027 policy requires. The two theorems carry proofs, the designs were preregistered, and every number in the submitted paper is produced by the supplementary code from pinned artifacts (the two discussion-period diagnostics by two further scripts under the same conventions, included in the revised supplementary material).
If there is a specific claim, proof, or number you believe is wrong, we will address it.