acceptodds
Under review as a conference paper at ICLR 2027

When Formatting Determines Rankings: Separating Semantic Accuracy, Compliance, and Strict Accuracy

Abstract

Language-model evaluations can conflate correctness, format adherence, and parser recognition. Across five models, MATH-500 and a balanced 13-domain MMLU-Pro subset, we vary final serialization while preserving the reasoning instruction and report extracted-answer accuracy, terminal-grammar compliance, and strict interface accuracy. The original 24,750 generations comprise a 17,250-response main experiment and a paired token-budget study. Paired intervals and multiplicity correction identify 12 strict-accuracy and 5 compliance rank reversals, but no reliable extracted-answer accuracy reversal. Nine original strict reversals persist under a wrapper-tolerant policy. A targeted 2,000-response, same-session intervention removes angle brackets around the answer placeholder, eliminating the Qwen32–DeepSeek-V3.2 Final/JSON strict point-order reversal on both tested subsets. A further study yields 3,000 outputs on held-out questions: complete prospective MMLU-Pro results support the prespecified model-by-format template interaction, while MATH results after an additional recovery request constitute a protocol-deviation sensitivity analysis, not overall prospective confirmation. Template effects are heterogeneous, including substantial accuracy declines in some conditions, and do not establish a universal repair. These findings demonstrate sensitivity of specified model–protocol snapshots, not invariant model deficiencies or latent capability ordering. Evaluations should report extracted-answer accuracy, compliance, and strict accuracy separately, alongside prompts, extraction rules, and token budgets.

Primary area:
datasets and benchmarks
Keywords:
large language models, evaluation, output formatting, answer extraction, ranking robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JLMEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors asked five models the same math and multiple-choice questions but changed how the final answer had to be written. Whether answers were correct did not changed model rankings, but whether answers were correct AND correctly formatted reversed rankings 12 times. The authors suggest that benchmarks should report correctness and format compliance separately to disentangle this observation .

Critical strengths

This conducts careful and through investigation with paired comparisons, multiplicity correction, frozen scorers, no manual overrides, and full release of the raw outputs. The findings to report correctness, compliance, and strict accuracy separately are practical, sensible and easy to follow through.

The authors also share results that go against their proposed idea honestly which helps readers make more informed decisions and interpretation, e.g. Qwen32’s sharp drop in performance under the revised prompt and the incomplete MATH confirmation

Critical weaknesses

The paper appears more like a rigorous technical report, where the novelty and scope of the investigation is a bit thin for a main track paper. On this front, a stronger argumentation and a richer set of analyses and discussions could help strengthen the intellectual framing to inspire deeper resarch questions and discussions. 

This paper is also farily dense and lacks narrative writing to make it more human readable.

Questions for authors

How much of the strict-reversal count remains if you only use the bracket-free prompt from the start? Was there analysis to show why or how models differ in compliance?

How model-dependent are the findings? Do the reversals appear with current frontier or reasoning models? From your observations, which model properties predict format compliance?

Reviewer_XnG2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript investigates the impact of output formatting on evaluation scores and rankings for LLMs. By varying the final answer format across five models and two benchmarks, authors find that strict accuracy rankings can undergo statistically significant reversals, while no statistically reliable reversals in semantic accuracy rankings are detected.

Critical strengths
  1. The research question is clear and practically relevant. The paper examines how output formatting affects model scores and rankings, helping improve the reliability of LLM evaluation.
  2. The experimental design is systematic, incorporating paired comparisons, multiple-comparison correction, and sensitivity analyses.
Critical weaknesses
  1. The generalizability of the findings remains limited, as the experiments cover a limited range of models, tasks, and output formats.
  2. The mechanisms underlying ranking reversals and differences across models remain insufficiently explained,lacking a systematic attribution analysis.
Questions for authors

A systematic attribution analysis is necessary to identify the key factors driving ranking reversals and to explain the observed differences in sensitivity across models.

1 comment

Thank you for your response.

Reviewer_qJ4aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the problem of when fixing the reasoning instruction, input questions, hyper-parameters, and graders, how much can changing only the required final output serialization impact the scores and model rankings?

They vary the formats of the answer output (they ask llms to give the final answer in 3 different ways/formats, such as JSON, or \box{<answer>}) and the max token length, compute the semantic accuracy and strict protocol accuracy, show how the ranking of the final models can change even reverse just based on the output format.

The experiment was done across 5 models, 2 benchmarks and 2 max token length budgets. They recommend people to report extracted-answer accuracy, format compliance, strict accuracy, and truncation separately.

Critical strengths

This paper studies an important and practical question in a well controlled way. Across three final answer serialization settings, they reported semantic answer accuracy, format compliance, and strict protocol accuracy. Overall, I find the experimental setup rigorous and it has practical value for people who run similar evaluations in practice.

Critical weaknesses

I find it's hard for me to appreciate the novelty of this work. I think this is a solid methodological case study: a well-documented audit of how prompt choices can impact evaluation results, with useful reporting recommendations. But its scientific novelty and generality are modest.

Questions for authors

Why do you think in figure 1, the R1-Distill-7B has such a low compliance rate? did the authors actually look into the generations of this model on the MATH dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.