acceptodds
Under review as a conference paper at ICLR 2027

Conservation-Aware Fine-Tuning: Injecting Physical Laws into LLM Reasoning via Preference Optimization

Abstract

Large language models frequently produce physically implausible reasoning—violating conservation laws of energy or momentum—even when their final numerical answers are correct. We introduce Conservation-Aware Fine-Tuning (CAFT), which treats conservation-law satisfaction as a programmatic preference signal for Direct Preference Optimization (DPO): a rule-based verifier labels each sampled reasoning chain as conservation-respecting or violating, and the resulting preference pairs fine-tune the model to prefer physically valid reasoning. The effect is scale-dependent rather than uniform. At 1.5B, CAFT reduces the Conservation Violation Rate (CVR) by 77.8% (5.62%1.25%, ; scenario-clustered ) and raises accuracy by 16.9 pp (40.31%57.19%, ) on 80 mechanics scenarios; the CVR reduction replicates directionally across three training seeds (1.25%/2.19%/3.75%), while the accuracy magnitude does not. At 8B no arm separates from the base model under our strip-first evaluation protocol, which removes hidden thinking content before verification (all arms at 0/320 violations), so we report the 8B scale as a boundary rather than a transfer. At 32B the base model already sits at the violation floor (0.31%) and CAFT changes nothing measurable ()—a ceiling effect with scale and model family co-varying. These three points distill into a deployment heuristic: apply CAFT only when the base model's CVR confidence interval excludes zero—a rule that is itself only as trustworthy as the protocol measuring CVR. Ablations attribute the conservation benefit to synthetic injection pairs, the accuracy benefit to wrong-answer pairs, and show the benefit is family-local; and a second, independently validated constraint probe shows no measurable side effect.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
physics-informed reasoning, conservation laws, preference optimization, large language models, verifiable supervision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SfW8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the problem of large language models (LLMs) violating physical conservation laws during reasoning, even when producing correct final answers. To address this issue, the authors propose Conservation-Aware Fine-Tuning (CAFT), which employs a rule-based conservation-law verifier to detect physical violations in reasoning chains and constructs preference pairs using four strategies to optimize the model's reasoning process through Direct Preference Optimization.

Critical strengths

1.CAFT achieves substantial improvements in answer accuracy on the 1.5B model under sampled decoding, demonstrating its effectiveness within the evaluated setting. 2. The paper includes extensive ablation studies and controlled experiments, providing useful evidence for assessing the effectiveness and limitations of the proposed method.

Critical weaknesses

1.The proposed method relies heavily on a conservation-law verifier, which primarily detects predefined patterns of incorrect physical statements. Consequently, the problem addressed by this paper is largely limited to specific verbalized physics misconceptions, rather than general physical reasoning correctness.

Notably, both the data construction statistics and experimental results indicate that the base model already exhibits a relatively low rate of violations detected by this verifier. Nevertheless, approximately 64.5% of the DPO preference pairs are constructed by artificially injecting predefined violations. In other words, the authors devote substantial synthetic supervision to teaching the model to disprefer error patterns that are already infrequently observed in its natural outputs. Given that the overall framework mainly combines existing techniques, including rule-based verification, synthetic error injection, and standard DPO, while addressing a relatively narrow problem, I find the methodological novelty and potential impact of this work limited.

2.There is substantial overlap between the training supervision and the primary evaluation metric. Specifically, the authors first construct rejected responses by injecting predefined violation patterns, use the corresponding rule-based verifier to construct preference pairs, and then evaluate the fine-tuned model using the same verifier. Consequently, the observed reduction in Conservation Violation Rate (CVR) may partly reflect the model learning to avoid specific predefined expressions, rather than a genuine improvement in its physical reasoning capabilities.

Furthermore, the so-called out-of-distribution (OOD) evaluation still relies on the same eight problem templates, with only numerical parameters changed. The paper does not sufficiently clarify the actual distributional differences between the in-distribution (ID) and OOD test sets. Therefore, it remains unclear whether the OOD evaluation demonstrates meaningful generalization beyond the problem distributions already covered by the training data.

Questions for authors

1.As I understand it, one of the main motivations of this work is to address cases where an LLM produces a correct final answer but makes physical errors during its reasoning process. Could the authors evaluate CAFT on a widely used public physics reasoning benchmark and use independent LLM evaluators to assess the physical correctness of the generated reasoning? In particular, it would be valuable to specifically examine responses with correct final answers but physically incorrect reasoning steps.

2.What are the specific differences between the ID and OOD test sets?

3.Why does CAFT show substantial improvements under sampled decoding but worse performance under greedy decoding? Could the authors provide further analysis and explanation for this discrepancy?

Reviewer_AmbZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAFT constructs DPO preference pairs from a lexical conservation-law verifier over generated mechanics reasoning chains. The pairs include natural positives/negatives, synthetically injected violations, same-answer pairs that differ in process validity, and correct-vs-wrong-answer pairs. On Qwen2.5-1.5B under sampled in-distribution decoding, CAFT reduces the verifier-defined CVR from 5.62% to 1.25% and improves answer accuracy in the headline seed; the CVR direction replicates across three training seeds, while the accuracy magnitude is unstable. Qwen3-8B/32B and a Qwen3-1.7B control are at or near the measured violation floor. The paper includes extensive ablations, matched-budget controls, clustered statistical tests, verifier audits, and explicit negative results.

Critical strengths
  1. The empirical reporting is transparent. The paper discloses null results, greedy-decoding failures, seed variance, model-family confounding, multiplicity, and verifier errors rather than hiding them.

  2. The controls are thoughtful: matched-budget outcome-only DPO, step-matched SFT, prompt-only and test-time-selection baselines, scenario-clustered tests, and same-answer-process pairs all help isolate what the preference signal changes.

Critical weaknesses
  1. The verifier does not verify conservation equations or recompute balances; it lexically matches nine misconception templates. Against a broader blind-audited physics-defect construct, precision is only 67.9–69.6% and recall 38.8–40.0%. Because the rejected chains are also generated from these templates, the main result is most directly evidence of suppressing templated verbalized misconceptions, not of injecting physical laws or improving physical reasoning.

  2. The positive result is confined to Qwen2.5-1.5B, in-distribution, sampled decoding. Greedy CVR is nominally worse, OOD conservation does not improve significantly, and Qwen3-1.7B/8B/32B are at the measured floor. The OOD split changes numerical values within the same eight templates. These results do not support the broad title or the proposed deployment rule from a single positive model-family/scale case.

Questions for authors
  1. Does the effect survive evaluation by an independent symbolic/numerical balance checker and blinded human physics experts on naturally generated, non-templated chains?

  2. Does CAFT generalize to unseen linguistic forms and structurally OOD, open-ended multi-step physics problems rather than new numbers in the same templates?

  3. If the claim is narrowed to suppression of templated verbalized misconceptions, what advantage remains over simpler targeted data augmentation or unlikelihood training?

Reviewer_URv3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper mainly investigates whether automatically identifying conservation-law violations in model-generated solutions can help train language models to produce more physically sound reasoning and improve their physics reasoning capabilities. The proposed method, Conservation-Aware Fine-Tuning (CAFT), constructs training data using automatically generated problems and a rule-based verifier that detects a predefined set of errors. On a 1.5B model, CAFT outperforms a standard SFT baseline under the paper’s sampled-decoding setting. However, the evidence presented does not establish that these gains extend to greedy decoding or larger models.

Critical strengths

The paper provides a relatively thorough disclosure of negative results. The idea of improving model capabilities by correcting conservation-law violations has some value.

Critical weaknesses

The evidence for the method’s generalizability is limited, and physics reasoning problems have largely been resolved in larger models, limiting the method’s practical value.

Questions for authors

1.Why do the gains not extend to greedy decoding in the paper's setting? On the 1.5B model, CAFT decreases greedy-decoding accuracy and slightly increases the violation rate. Could a temperature sweep and per-problem error analysis explain this discrepancy and clarify which decoding settings benefit from the method? 2.Has the SFT baseline been sufficiently optimized? Its performance actually declines under the standard sampling setting. Could the SFT degradation stem from data-selection bias or the training configuration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.