acceptodds
Under review as a conference paper at ICLR 2027

Does Post-Training Tax Exact Arithmetic? It Depends on the Route

Abstract

Does post-training reduce accuracy on exact arithmetic, or can the evaluation interface change that conclusion? We study 16 released checkpoint pairs on six integer-arithmetic tasks, including seven documented post-training comparisons from three pretrained bases. We first compare checkpoints using the same completion input, without a chat template. We then compare completion and chat inputs for the same post-trained checkpoint. The common comparison between base completion and post-trained chat changes both checkpoint and interface. In a pre-registered replication that constrains outputs to integers, the common and fixed-completion comparisons give opposite conclusions in 24 of 96 pair–task combinations: one indicates significantly higher accuracy and the other significantly lower accuracy after Holm correction. In tokenizer-compatible pairs, the size of the checkpoint effect depends on the input interface. Two selected problem subsets show reversals under both constrained decoding and free generation with first-integer scoring. This does not establish independence from generation and scoring conditions. Some deficits are no longer detected after changing the prompt, whereas others persist. For OLMo-3, we ask subtraction questions without a chat template and allow free generation, scoring the first integer. On 14-digit problems, accuracy is 52.4% for the base checkpoint and 1.4% for the final checkpoint. Among answers that are correct at the base checkpoint but wrong at the final checkpoint, most first become wrong at the supervised fine-tuning stage. For Llama-3.1/Tulu, gains measured with the same completion input become negative estimates under the question form. A secondary transplant of base-checkpoint states corrects some selected expression-form errors but none of ten tested 14-digit subtractions. These findings concern accuracy under specific evaluation conditions. Comparisons of post-training should distinguish checkpoint, interface, and answer-extraction conditions before attributing score differences to changes in arithmetic ability.

Primary area:
datasets and benchmarks
Keywords:
post-training, evaluation methodology, exact arithmetic, chat template, constrained decoding, alignment tax, language models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Hr8sOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper analysed post-train tax in the exact arithmetic area. The authors claimed that post-training comparisons must specify interface, decoding, and answer-extraction conditions rather than equating score changes with route-independent arithmetic ability.

Critical strengths
  1. Clear and useful evaluation design. Separating checkpoint, interface, and bundled effects makes base–post-trained comparisons more interpretable.
  2. Interesting finding that some fail first at SFT and remain wrong.
Critical weaknesses
  1. The paper is hard to follow.
Questions for authors

None.

1 comment

Response to Reviewer Hr8s

Thank you very much for the review and for the encouraging comments on the evaluation design. You are right that the paper was hard to follow, and we apologise for the difficulty. We rewrote the Introduction in plain words: it now introduces only the terms needed to state the question and the main result (the two input formats, the evaluation setup, the three comparisons of checkpoint, input format, and both combined, and what we call a reversal), followed by a motivating example, and every other term is defined in the section where it is first needed. Please also see the general response for the new audit without constrained decoding.

Reviewer_ppnzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how post-training affects exact arithmetic performance under different evaluation conditions. The authors compare base and post-trained checkpoints using shared completion and native chat formats across 16 checkpoint comparisons and six arithmetic tasks. A pre-registered replication identifies 24 of 96 task-level comparisons where the two evaluation paths yield opposite statistically significant conclusions. Additional analyses examine prompt sensitivity, training-stage differences, and activation transplantation.

Critical strengths
  1. Extensive empirical validation. The study evaluates multiple checkpoint families and arithmetic tasks, with paired comparisons, multiple-testing correction, and pre-registered replication.
  2. Interesting stage-level observations. The OLMo-3 experiments identify substantial degradation in long-digit subtraction that persists under question-form free generation, with most losses first appearing at SFT.
Critical weaknesses
  1. Limited novelty and explanatory contribution. Prior research has already demonstrated that base-versus-instruct comparisons depend on prompting, chat templates, and evaluation procedures. The main contribution primarily quantifies this phenomenon in exact arithmetic. Moreover, the documented post-training comparisons involve only three distinct pretrained base models, without systematically comparing different SFT or preference-optimization methods. Consequently, the study provides limited insight into which aspects of post-training cause arithmetic degradation or why.
  2. Insufficient evaluation-condition coverage. The primary experiments rely on the specific "Output ONLY a number." instruction, grammar-constrained decoding, and a limited chat configuration. These choices may substantially influence the observed differences. The paper's own analyses show that changing prompt formats, number serialization, decoding, or answer extraction can alter effect sizes and even directions. Only two of four selected slices retain reversals under both constrained and free-generation conditions. A more systematic evaluation across different prompt formulations, native chat settings, and output constraints is needed to establish the generality of the reported 24/96 reversals.
Questions for authors
  1. How many of the 24 reported reversals persist under different prompt formulations and unconstrained generation with appropriate answer extraction?
  2. What new understanding of post-training does this study provide beyond previously established prompt- and template-dependent performance differences, particularly without systematic comparisons of training methods?
  3. Can the authors distinguish the effects of chat-template formatting from those of the numeric-only instruction, constrained decoding, and stopping conditions? How would this affect the interpretation of the reported route effects?
2 comments

Response to Reviewer ppnz

Thank you very much for the careful review and for recognising the paired design, the multiple-testing control, and the pre-registered replication. We agree that the paper builds on earlier work, and would like to clarify what it adds. Earlier work showed that base-versus-instruct comparisons depend on prompts and templates; what had not been measured, to our knowledge, is how often that dependence reverses a multiplicity-corrected conclusion, replicated on new items under a pre-specified analysis. Here it does so in 24 of 96 pair–task combinations, and 19 of the 24 remain without constrained decoding (first-integer scoring).

  • "Limited novelty and explanatory contribution." Exact arithmetic makes this measurable: every item has one checkable answer and difficulty is set by digit length. The 24 survive one correction over all 288 tests, and 23 of the 32 discovery reversals recur among them on item-disjoint grids (Section 4.3, Appendix D.1). Limit: the seven comparisons come from three pretrained bases, training methods are not compared, and the analyses reported here do not identify the cause of the OLMo-3 loss (Sections 3.3, 4.5, 5).
  • "Insufficient evaluation-condition coverage." We agree that broader coverage would strengthen the paper, and your question prompted the new audit. Within one study we fixed one setup in advance, and we state conclusions separately for each evaluation setup reported here; the new audit removes the decoding constraint on all 24 reversals, 2,000 items each (Section 4.3, Appendix F.7).
  • Q1 (how many persist). Without the constraint, 19/24 remain under first-integer scoring, 15/24 when any integer equals the answer, and 11/24 under the last integer, on the same items (Table 23). They show three different failure patterns; the analyses reported here do not identify their cause, and we do not read them as differences in arithmetic ability. Limit: among the analyses reported here, only Tulu final subtraction was re-measured under the question form (on original-grid subsets), where the two comparisons no longer point in opposite directions (Section 4.4); an earlier check also tried a 3-shot base prompt on four of the 24 (Appendix F.6). The paper reports no prompt sweep over the 24.
  • Q2 (new understanding of post-training). Beyond the frequency above, the design separates losses no longer detected under a changed prompt (OLMo-3, 3 digits) from a persistent one where 253 of 259 lost 14-digit answers first fail at SFT (Section 4.4). We do not claim which training method causes it.
  • Q3 (template vs. instruction, decoding, stopping). In part: the chat template is isolated as a whole at a fixed checkpoint (Section 3.2); the numeric-only instruction was held constant; decoding was varied with the stop set kept (Appendix F.1: output constraint removed) and with both removed (Appendices F.4, F.7); these analyses cover different pair–task combinations and items, so the separate effect of stopping on the 24 reversals is not identified.

We would be glad to clarify any remaining point.

Reviewer_ppnzOpenReview ↗

read the author's response

I have read the authors' response.

Reviewer_mdsAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They study whether post training taxes llm’s ability to solve arithmetic problems. They evaluated models on 6 arithmetic tasks, from add, sud, multiplication by 1, 2, and d digits, and division.

For both base models and post trained models, they run them in base format (direct next word generation) and with chat template, which they call the routes.

They found that the results can be different even opposite for different routes, and argue that future evaluation should report checkpoint and route comparisons separately, alongside the prompt, decoding, stopping and answer extraction rules.

Critical strengths

Testing both base model and post trained models, with and without chat template is a good control. Experiments on intermedia checkpoints, such as SFT, DPO, are useful.

Critical weaknesses

I find the contribution of this paper limited. The authors showed that you can get opposite evaluation results when evaluating base and post-trained models on arithmetic problems, and post-training doesn’t always hurt the base models’ ability on this task, but where do we go from there? Changing the prompt format from expression to question changes the accuracy dramatically, which makes the reader question how much we can trust the evaluation results — are the results and findings mostly artifacts of prompt format or answer parsing and extraction rules?

Questions for authors
  1. what does this mean, line 201: Capacity-aware redistribution leaves 39 of 96 fresh cells with three observed points and 78 without a digit-2 row (Appendix D)

  2. do you have any insights about why add, sub and digit 1 multiplication so hard for post trained OLMO3 models? In figure 2, the tax mostly happened to OLMO3 models, is that a model specific artifact?

1 comment

Response to Reviewer mdsA

Thank you very much for the positive assessment of the completion/chat control and the intermediate-checkpoint experiments, and for the concrete questions. On the contribution, we would like to explain why we believe it matters for anyone reading a base-versus-instruct comparison. In our pre-registered replication, in 24 of 96 pair–task combinations the usual comparison and the fixed-format comparison give opposite corrected conclusions, and all 24 survive a single correction over all 288 tests (Section 4.3).

  • "Where do we go from there?" A concrete reporting rule: compare the base and post-trained checkpoints using the same completion format, and separately compare completion and chat formats at the post-trained checkpoint; report both next to the usual combined number, with the prompt, decoding, stopping, and extraction rules (Section 6).
  • "Artifacts of prompt format or answer parsing?" This is a fair question. In our view, sensitivity to the evaluation setup is what the paper measures, not a flaw of the measurement; the design shows which effects change with the setup and which persist. Under constrained decoding, strict and tolerant extraction give the same signs and significance decisions in all three 96-combination families (Appendix D). OLMo-3's 3-digit subtraction loss is no longer detected under a question form, while its 14-digit loss persists (52.4% base vs. 1.4% final; Section 4.4, Table 2). Without constrained decoding, 19/24 reversals remain (see the general response).
  • "Line 201." We apologise for the compressed sentence. Replication items had to be new and two-digit problem pools are finite, so the pre-specified rule moved the shortfall to other digit widths, keeping 2,000 items per combination; this removed the two-digit width in 39 of 96, and another 39 never had one. Restricting to widths seen in both grids leaves all 24 reversals unchanged (Section 3.1, Appendix D).
  • "Why so hard for post-trained OLMo-3; model-specific?" In our data, yes: every significant fixed-format loss among the seven post-training comparisons is in OLMo-3 (Section 4.1, Figure 2), and 630 of 708 lost subtraction answers first fail at SFT (Appendix G, Table 25). Limit: the analyses reported here do not identify the cause, three pretrained bases cannot separate recipe, data, and base, and the paper does not claim a general post-training tax (Section 5).

We would be glad to clarify any remaining point.

Reviewer_MgyFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of whether post-training degrades arithmetic performance on addition, subtraction, several multiplication tasks (multiplying by one digit, two digits, or the same number of digits as the first argument), and exact division (without remainder). The paper attempts to separate the effects of two different aspects: First, whether the checkpoint is the pre-trained "base" checkpoint or if it is a post-trained release checkpoint (e.g., after SFT). Second, the "route", which I understand to be the prompt formatting (e.g., do we simply give the model a sequence to complete, or should we include things like change of turn tokens, tokens for delimiting thoughts, etc.).

The authors discover that for some pairs of pre- and post-trained checkpoints, using a completion-style prompt recovers some accuracy in arithmetic tasks that the chat-based prompts lose (under constrained decoding where the model sampling is limited to the set of tokens consistent with a grammar describing the output format). In some cases, removing the constrained decoding improves accuracy for the chat-formatted prompts again. There are a number of additional supporting experiments (including changing the prompt format, and experiments around "base-content transfer" where the post-trained model's activations are modified at runtime to try to recover pre-trained performance).

Critical strengths

I think the main strength of the paper is that the findings are somewhat surprising. My intuition suggested that in essentially every case, it would not make sense to evaluate the post-trained model using the completion-formatted prompts, since these are now out of distribution. It was surprising to me that for some of the pre- and post-trained checkpoint pairs that completion-formatting prompts actually produced higher accuracy than the chat-formatted prompts.

Having said that, it is important to note that these experiments use a grammar-constrained generation, where the model is only permitted to output tokens consistent with a grammar (even when those tokens have very low probability mass in the next-token probabilities). This could potentially be responsible for many of the findings, since it seems likely to force models to use tokens they would not otherwise use. This seems especially likely for chat-tuned models that have a tendency to start their response conversationally, instead of immediately responding with the answer.

Critical weaknesses
  1. The paper is written in a way that is very difficult to read. Many terms and concepts are used without introduction, which makes several relatively clean ideas difficult to pull from the text. I think the clarity issues are severe enough that the paper is not ready for publication, even without the other weaknesses.
  2. I think constraining the model outputs so that they must conform to a specific grammar may force many of the post-trained models to output tokens that they would almost never output without the constraint. I don't see why their constrained behavior should be representative of their actual skill at the arithmetic tasks. There is some exploration of free-generation, but most of the findings are in the constrained setting. It seems like a more principled approach would be to be lenient with the scoring (for example, grading the model's output as correct if it includes a single number that is equal to the actual answer, even if it output other tokens).
  3. I feel like trying to disentangle the effects of prompt formatting from post-training is potentially misguided. For example, I think most models are trained in such a way that the change of turn formatting tokens are literally never seen during pre-training, so it would be surprising for the pre-trained model to behave reasonably when they are present. For the post-trained model, if it were fine-tuned to do arithmetic in base 16 or to speak another language, it seems like measuring it's performance on base 10 or in English would be less principled than adjusting the prompt format to match the fine-tuning.

Small sample of clarity issues:

  • Line 53: What is a "fresh-grid" item?
  • Line 53 and 77: What are "fixed-completion checkpoint difference[s]" and "bundled difference[s]"?
  • Line 77: What does Holm-significant mean? (i.e., I assume it is related to the Holm–Bonferroni method, but this should be clearly stated).
  • Line 82: What is the "measurement route" and what does it mean to make it part of the estimand? If I understand correctly, by "measurement route" you mean how the prompt to the model is formatted, and you intend to separate the prompt formatting effects from the post-tuning.
  • Line 83: What is the "fixed-route checkpoint contrast", the "route contrast at a fixed post-trained checkpoint", and "bundled contrast"? I think I can guess the meanings of these terms after reading more of the paper, but they should be introduced before they are used.
Questions for authors

Do you think if you redid the main experiments without constraining the model output tokens and allowing for a more lenient accuracy measure, would the results stay the same?

1 comment

Response to Reviewer MgyF

Thank you very much for the detailed and thoughtful review, and in particular for listing concrete clarity problems line by line and for raising the validity question directly; both improved the paper. We address your points below. On the main one, we would respectfully note that constrained decoding is an established technique used in deployed structured-output systems, and restricting the answer space is routine in evaluation, so accuracy under output constraints is a practically relevant evaluation target in its own right. We do not claim that our integer-output setting represents unconstrained arithmetic ability, and we agree that the constraint can change which tokens a model produces, which is why the revision tests the reversals without it.

  • "Would the results stay the same without constraining the output and with a more lenient accuracy measure?" Partly. Without the constraint, on the same 2,000 items each, how many reversals remain depends on the scoring rule: 19/24 with first-integer scoring, 15/24 with the any-integer rule you suggested, and 11/24 with last-integer scoring (Section 4.3; Appendix F.7, Table 23). A post-hoc check that handles decimals strictly gives 16/24 for the any-integer rule and 10/24 for the last integer. Limit: this covers the 24 selected pair–task combinations on the same items and changes only decoding and scoring; prompt form and chat settings are unchanged.
  • "Constrained behavior may not represent actual skill." The remaining reversals show three different failure patterns (chat output that stops at once, reasoning cut off by the length limit, completion output that is never correct); the analyses reported here do not identify their cause, and we do not read them as differences in arithmetic ability (Appendix F.7, Table 24).
  • "Many terms are used without introduction." You are right, and we are grateful for the specific line references. We rewrote the Introduction in plain words and moved every definition it does not need to the section where the term is first used. The terms you listed are now defined before first use: "measurement route", "estimand", the three comparison names, the fixed-completion and bundled differences, and "Holm-significant" in Section 1; "fresh grids" in Section 3.1.
  • "Disentangling prompt formatting from post-training is potentially misguided." We appreciate this concern and agree with its premise: the two input formats (plain completion vs. chat template) cannot be matched. We would respectfully suggest that this is exactly why they need to be separated. Common practice compares a base model under completion with its chat model under the chat template and reads the difference as the effect of post-training; that single number mixes two changes. Your intuition is, in our view, the reason a fixed-format comparison is needed: base accuracy through the chat template is at or below 1% in 40 of the 54 combinations where it was run (Section 4.2, Figure 3). Nor is the chat format uniformly favourable to the chat model: among the 24 reversals its effect ranges from +65.25 to −64.30 pp (Appendix D.1, Table 14). We do not claim either format is a neutral measure, so we report the three comparisons separately.

We would be glad to clarify any remaining point.

Comments on the paper

1 comment

General response to all reviewers and the AC

We sincerely thank all four reviewers and the AC for their time and for careful, constructive reviews. Two concerns were shared across reviews, and the revision addresses both.

1. Clarity (MgyF, Hr8s; presentation scores of mdsA and ppnz). The reviewers are right. We rewrote the Introduction in plain words. It now introduces only the terms needed to state the question and the main result (the two input formats, the evaluation setup, the three comparisons, Holm-significance, and what we call a reversal), explains the comparisons in words before any notation, and gives the motivating example right after them; every other term is defined in the section where it is first needed. The terms Reviewer MgyF listed are all defined before first use, and we replaced our coined word "contract" with "evaluation setup" throughout.

2. Constrained decoding and scoring (MgyF, mdsA, ppnz). Constrained decoding is an established technique used in deployed structured-output systems, so accuracy under output constraints is a practically relevant evaluation target in its own right. Our integer-output setting is one explicit condition for comparing exact-answer accuracy; we do not claim that it represents every structured-output application or unconstrained arithmetic ability. We agree that the constraint can change which tokens a model produces, so we tested whether the reversals also occur without it. A reversal is a pair–task combination where the usual comparison and the fixed-format comparison reach opposite significant conclusions. On all 2,000 replication items of each of the 24 reversals, with the numeric constraint and the registered stop set removed and the analysis pre-specified, the reversal remains in 19/24 under first-integer scoring, 15/24 when any integer in the output equals the answer, and 11/24 under the last integer (Section 4.3; Appendix F.7, Tables 23–24).

Limits: the audit uses the same items, not new ones. It changes only decoding and scoring; prompt form and chat settings are unchanged. The remaining reversals show three different failure patterns; the analyses reported here do not identify their cause, and we do not read them as differences in arithmetic ability.

The abstract now reports this re-evaluation in one sentence, replacing the sentence on the earlier four-subset check; all previously reported numbers are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.