acceptodds
Under review as a conference paper at ICLR 2027

When Math Becomes Physics, Can LLMs Keep Up?

Abstract

Can a language model solve a physics problem as reliably as the mathematics underlying it? We introduce Math2Physics, a benchmark of 216 mathematics–physics pairs from six collections, linked by shared central derivations. An expert-guided conversion procedure expresses mathematical constraints through physical laws, observables, and measurements, producing paired tasks that probe the recovery of mathematical structure from physical descriptions. Expert-reviewed annotations characterize physical domain, authenticity, and the added interpretation steps. Across three models, eight reasoning settings, and 64 attempts per question, mathematics accuracy is 77.03% and physics accuracy is 71.58% on the common 212-pair panel: a 5.45 percentage-point gap (95% paired-bootstrap interval ). The disadvantage recurs on the same problems across models, and greater reasoning effort does not consistently close it. Yet oracle coverage reaches 93.40% for physics at 64 attempts, approaching mathematics at 94.00%. This contrast reveals a gap between finding a correct physics solution among many attempts and producing one reliably in a single attempt. We further use the paired response banks to compare mathematics and physics preference training on held-out physics problems. Math2Physics provides a common foundation for studying scientific reasoning across formulations, reasoning settings, and training subjects.

Primary area:
datasets and benchmarks
Keywords:
large language models, mathematical reasoning, physical reasoning, cross-domain transfer, paired benchmarks, synthetic data generation, direct preference optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tLYxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the performance of current LLMs on math problems and their equivalent physical problems. It introduces Math2Physics, a benchmark containing 216 problem pairs, and a math-to-physics skill for converting math problems into physics problems. Experiments show that models generally perform worse on physics problems than on their mathematical counterparts, but this gap narrows substantially as the number of sampling attempts increases.

Critical strengths
  1. The paper reveals a performance gap between current LLMs’ ability to solve physics problems and their ability to solve equivalent mathematical problems.

  2. It proposes a math-to-physics skill that may be useful for the community to synthesize physics data.

Critical weaknesses
  1. The Math2Physics benchmark is too small and too easy to clearly distinguish the capabilities of frontier LLMs.

  2. The paper lacks an analysis of the performance gap. It only reports the existence of the gap without explaining its causes.

  3. The gap narrows to 0.6% with 64 attempts, suggesting that it may arise from sampling randomness rather than a systematic capability gap. This limits the significance of the this research.

  4. The purpose of the DPO experiment is unclear. Post-training LLMs on physics data would naturally be expected to improve their performance, but this experiment does not seem to contribute to the paper’s overall analysis or conclusions.

Questions for authors

I am curious about the detailed implementation of the math-to-physics skill. How does it transform an abstract mathematical problem into concrete physics problems across different domains?

9 comments
AuthorsOpenReview ↗

W1. Benchmark Size and Difficulty, and W2. Gap Analysis (1/3)

W1. The Math2Physics benchmark is too small and too easy to clearly distinguish the capabilities of frontier LLMs.

Could you clarify how many problems you would consider sufficient and what level of difficulty you would not consider "too" easy?

Math2Physics contains 216 expert-written/reviewed mathematics-physics pairs. The mathematical datasets we use are common in the community; for example, MathArena reports accuracy for Qwen3-235B-A22B, DeepSeek-R1-0528, GLM 4.5, Kimi K2, gpt-oss, and many other models on SMT 2025 and CMIMC 2025 [1]. Almost all model cards report performance on AIME. Harder datasets include, for example, IMO-ProofBench (Olympiad-level proof problems) [2], Riemann-Bench (25 research-level problems) [3], and HorizonMath (for mathematical discovery) [4]. These target specialized tasks in proof writing and research mathematics. Competition-math benchmarks such as AIME and HMMT recur across model-release reports and academic evaluations [5-9].

These harder benchmarks have specialized applications in developing Olympiad proof-writing systems and assessing progress toward mathematical discovery. These applications are outside the scope of our paper.

Therefore, we believe our choice of mathematical datasets follows common LLM evaluation practice in both size and difficulty.

In addition, using the skill we developed, anyone in the community can create their own physics datasets from existing mathematics datasets (Appendix D.2). We encourage the reviewer to read Appendix A, which shows how existing mathematics problems can be converted into physics problems while preserving the same logical structure. This transformation is a reusable framework that allows us to further expand the Math2Physics analysis by following the skill guidelines.

If the base dataset (the math part) is difficult, the resulting physics problems tend to be difficult too (Appendix B.7, Figure 20a). We also encourage the reviewer to look at the examples in Figure 3 and Appendix A.3 (Figures 9 and 12); these are not easy problems. We also refer the reviewer to our difficulty analysis in Section 3.4 and Appendix B.11 (Table 16).

W2. The paper lacks an analysis of the performance gap. It only reports the existence of the gap without explaining its causes.

We respectfully disagree. Here is why:

  1. We devote Section 4. The Math-Physics Performance Gap to analyzing the gap. We compare eight model and reasoning settings, including no thinking and different levels of reasoning effort (low, medium, and (x)high). We include both dense and MoE models to examine how performance changes across settings and models.

  2. Reasoning effort (Section 4.3, Figure 6; Appendix B.4, Tables 7-8). More reasoning does not consistently close the gap.

  3. Recurring problems across models (Section 4.4; Appendix B.5, Figure 19). Different models tend to show the gap on the same problems (correlations: 0.282-0.485).

  4. Mathematical difficulty (Section 4.4, Figure 7a; Appendix B.6, Table 9; Appendix B.7, Figure 20a and Table 11). The gap is 7.12 pp for Easy problems, 3.77 pp for Medium, and -0.95 pp for Hard.

  5. Physical characteristics (Section 4.4, Figures 7b-d; Appendix B.6, Tables 9-10). We examine the physics topic, how natural the scenario is, and the number of added physics interpretation steps.

  6. Repeated sampling (Test-time scaling) (Section 5.1, Figure 8; Appendix B.8, Figure 20b; Appendix C.1, Figure 21). When a problem counts as solved if any attempt is correct, the gap falls from 5.71 pp with one attempt to 0.80 pp with 64 attempts.

  7. Scoring and evaluation choices (Section 4.2; Appendix B.9, Table 12). The gap remains 5.71 pp when answers may omit units and is 10.02 pp under strict scoring.

  8. Performance across collections and settings (Section 4.2, Figure 5; Appendix B.3, Figure 18 and Tables 5-6). Leaving out any one collection leaves an overall gap of 4.99-6.72 pp.

References

[1] MathArena Team. MathArena, 2025.

[2] Luong et al. Towards Robust Mathematical Reasoning. EMNLP 2025.

[3] Garre et al. Riemann-Bench: A Benchmark for Moonshot Mathematics. 2026.

[4] Wang et al. Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification. 2026.

[5] Dekoninck et al. Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs. 2026.

[6] Qwen Team. Qwen3.6-35B-A3B. Official model card. 2026.

[7] Meta. Muse Glimmer-30B: Model Card and Evaluation Methodology. 2026.

[8] Microsoft. Phi-4-Reasoning Technical Report. 2025.

[9] NVIDIA. Nemotron-Cascade-2. Technical report. 2026.

AuthorsOpenReview ↗

W3. Repeated Sampling, and Question: Math-to-Physics Skill (2/3)

W3. The gap narrows to 0.6% with 64 attempts, suggesting that it may arise from sampling randomness rather than a systematic capability gap. This limits the significance of this research.

This comment seems to conflate two measures: single-attempt accuracy and oracle pass@k.

  1. Single-attempt accuracy (Section 4.2, Table 1). We estimate how often an individual answer is correct using 64 attempts per question; this is simply average accuracy. The median of mathematics-physics gap is is 6.27 percentage points. So yes, the gap is systematic as Table 1 shows.

  2. Oracle pass@64 (Section 5.1, Figure 8). This counts a problem as solved if any of 64 answers is correct. We further explain what narrowing the gap shows in our response to W4 (3/3).

This is actually not a weakness rather it is one of the strong observations of the paper: LLM physics performance is lower in single-trial accuracy, but it becomes close to math if models have more time/compute.

The revised Section 5.2 builds on this observation: post-training on the physics problems turns this coverage into single-attempt accuracy on held-out problems, by 4.11 pp in physics and 3.49 pp in mathematics.

Questions

I am curious about the detailed implementation of the math-to-physics skill. How does it transform an abstract mathematical problem into concrete physics problems across different domains?

Physics experts first converted 30 mathematics problems into physics counterparts while preserving the underlying mathematics. We distinguish two levels: a physical model of a phenomenon and the mathematical abstraction of that model. Starting from the mathematical abstraction, the experts construct a physical model that realizes it. We call this step "expert construction" (Figure 2). We then encoded the conversion procedure illustrated by the 30 CSAT mathematics-physics pairs as a set of instructions for LLMs.

For a new mathematics problem, an agent first solves it and records its variables, constraints, requested quantity, and essential solution steps. It then proposes three candidate physical models and selects one whose laws and measurements recover the same mathematical relationships. For example, optical reflection and travel times recover geometric constraints (Figure 9), while a volume integral becomes a moment-of-inertia calculation under a specified density (Figure 10). The agent then writes and solves the physics problem and checks the mathematics, physical laws, units, alternative interpretations, and shortcuts. The construction is retained if all checks pass.

In response to your question, we added an example of a skill-guided conversion in Section 3.2 (Figure 3) and four further examples in Appendix A.4. Please let us know if you have any further questions about the skill.

AuthorsOpenReview ↗

W4. Purpose of the Training Experiment (3/3)

W4. The purpose of the DPO experiment is unclear. Post-training LLMs on physics data would naturally be expected to improve their performance, but this experiment does not seem to contribute to the paper's overall analysis or conclusions.

We revised Section 5.2 to make its purpose and contribution explicit. The experiment tests the paper's main takeaway: physics counterparts of mathematics problems are a useful source of post-training data. Section 5.1 shows that correct physics answers are available for most problems (oracle coverage 93.11% at 64 attempts) but are produced less reliably on a single attempt. Section 5.2 asks whether post-training turns this coverage into single-attempt accuracy, and how physics data compares with mathematics data.

The revised study trains Qwen3.5-4B on either the physics or the mathematics form of the same 156 families and evaluates both forms of 60 held-out families against the untrained base model (Table 2). The results go beyond "physics data improves physics":

  1. Physics training improves both subjects: +4.11 pp in physics (95% CI [1.20, 7.08]) and +3.49 pp in mathematics ([0.62, 6.41]). Mathematics training improves mathematics (+4.06 pp) but adds only 0.78 pp in physics. The transfer to mathematics is consistent with the physics problems retaining the mathematical structure of their sources, which only the paired design can test: both models learn from the same families and differ only in formulation.
  2. Physics training narrows the gap the paper measures: mathematics training widens the base model's mathematics-physics gap from 5.10 to 8.39 pp, whereas physics training narrows it to 4.48 pp (3.91 pp less; CI [0.83, 7.19]) while improving mathematics as much as mathematics training does.
  3. Gains on hard problems: on Hard physics problems, accuracy rises from 7.81% to 12.89%, and the physics-trained model solves 2 of the 7 held-out physics problems that the base model never solves in 32 attempts.
  4. Responses become more reliable: budget-exhausting responses fall from 20.31% to 15.52%, repetitive loops from 10.83% to 8.13%, and responses become shorter.

These results connect the three parts of the paper: a fair measurement of the gap (Section 4), evidence that the gap concerns reliability rather than the availability of correct solutions (Section 5.1), and a demonstration that physics problems built with our skill improve single-attempt accuracy in both subjects and narrow the gap (Section 5.2). Per-collection results are in Table 19, and further analyses in Appendix C.5.

Reviewer_uUNvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Math2Physics, a paired benchmark comprising 216 pairs of problems that share a common mathematical core, with each pair consisting of a pure mathematics problem and a corresponding physics formulation. The paper then conducts several exploratory analyses using this benchmark. Its main finding is that physical formulations reduce single-attempt reliability, particularly for lower-performing models, as evidenced by accuracy measurements across multiple models. The paper also includes a small-scale DPO training experiment.

Critical strengths

The paired design is relatively clean and meaningful, helping distinguish the underlying mathematical difficulty from other factors involved in solving physics problems. The resulting dataset is a useful contribution, and the main finding—that models perform worse on the physics formulations—is supported by substantive empirical evidence.

Critical weaknesses

1.The results are largely unsurprising: reformulating mathematics problems as physics problems introduces additional interpretive and modeling requirements, so a decrease in accuracy is expected. However, the paper does not further disentangle the sources of this difficulty, and therefore provides limited additional insight. 2.The DPO experiments do not yield clear or conclusive findings.

Questions for authors
  1. How can the difficulty of physical reasoning be distinguished from a general increase in task complexity?
  2. What new insights does this benchmark provide beyond the observation that adding physical interpretation steps reduces accuracy?
9 comments
AuthorsOpenReview ↗

Request for Clarification on Review Revision

Dear Reviewer uUNv,

Your review was:

Critical Strengths:

The paired design is relatively clean and meaningful, helping distinguish the underlying mathematical difficulty from other factors involved in solving physics problems. The resulting dataset is a useful contribution, and the main finding—that models perform worse on the physics formulations—is supported by substantive empirical evidence.

Critical Weaknesses:

The paired design is relatively clean and meaningful, helping distinguish the underlying mathematical difficulty from other factors involved in solving physics problems. The resulting dataset is a useful contribution, and the main finding—that models perform worse on the physics formulations—is supported by substantive empirical evidence.

Questions For Authors:

The paired design is relatively clean and meaningful, helping distinguish the underlying mathematical difficulty from other factors involved in solving physics problems. The resulting dataset is a useful contribution, and the main finding—that models perform worse on the physics formulations—is supported by substantive empirical evidence.

However, you later changed it to the current version. Although reviewers can revise their reviews during the discussion period, we were wondering why you chose to revise your review even before we had submitted any response. Could you explain the reason for this change?

Dear Authors,

Thank you for raising this, and I apologize for the confusion. The initially submitted text was the result of a copy-paste error when transferring my review from my local draft to OpenReview: the same paragraph was mistakenly pasted into multiple fields. I noticed the mistake later and replaced it with my intended review, which is the current version. The revision only corrected this formatting error and did not reflect any change in my assessment.

The current version reflects my actual evaluation. I would welcome your response to the weaknesses and questions raised there, and I will carefully consider it in the discussion.

Best regards, Reviewer uUNv

AuthorsOpenReview ↗

Thank you and a quick question

Dear Reviewer uUNv,

We thank you for your response and clarification. While we are preparing responses to your questions, respectfully and just out of curiosity, we would like to ask whether your rating of 1 (clear rejection) was intended or not. We ask this question especially because we strongly believe your comments don't justify your extremely low score since your review was very positive:

The paired design is relatively clean and meaningful, helping distinguish the underlying mathematical difficulty from other factors involved in solving physics problems. The resulting dataset is a useful contribution, and the main finding—that models perform worse on the physics formulations—is supported by substantive empirical evidence.

While we would add "the gains in both physics and mathematics from physics post-training (Section 5.2), the finding that repeated sampling nearly closes the oracle-coverage gap (Section 5.1), and the reusable math-to-physics skill (Section 3.2)" to your points, even without that, you consider the contributions useful for the community, which makes your rating of 1 (clear rejection) difficult to understand. The comment above answers the questions that the ICLR reviewer guidelines ask: "Does the paper support the claims?" "What is the significance of the work?" and "Is the approach well motivated, including being well-placed in the literature?" (questions are excerpted from the ICLR 2027 reviewer guidelines).

This is not directly stated in the reviewer guidelines, but common practice is that if a paper has at least one nice contribution that could help the community, the rating leans toward acceptance. Otherwise, you may consider a rejection. But clear rejection means the paper has no contributions at all or has serious problems, such as a scientific error or plagiarism, etc. None of these is the case in your review. We just wanted to respectfully ask for your clarification.

Best regards,

The authors

Reviewer_533AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Math2Physics, a benchmark pairing mathematics problems with physical reformulations that share a central mathematical derivation. Evaluations across multiple language models and reasoning settings show that physical reformulation generally reduces accuracy, while increased reasoning effort does not consistently close the gap. Repeated sampling substantially narrows the oracle coverage gap, suggesting a distinction between generating a correct physics solution and doing so reliably.

Critical strengths
  1. The dataset construction is careful and auditable. Expert-written examples, multiple critique stages, physics-expert review, and documented correspondences support inspection of physical validity and fidelity to the mathematical sources.

  2. The evaluation covers multiple models and reasoning settings. Repeated sampling, paired statistical analyses, and multiple-comparison correction provide stronger support for the main findings than single-generation or single-model evaluations.

  3. The paired design helps control for differences in mathematical content. Comparing mathematical and physical formulations that share a central derivation provides a more direct assessment of formulation effects than comparisons between unrelated subject-specific benchmarks.

Critical weaknesses
  1. The conceptual novelty of the main finding is limited. An accuracy decrease after replacing explicit mathematical conditions with physical descriptions requiring additional knowledge and interpretation is broadly expected. The paper does not sufficiently establish what new understanding follows from this observation.

  2. shared central derivation does not guarantee full task equivalence. Some conversions alter the requested quantity or source interpretation, complicating attribution of the aggregate gap to formulation alone. A separate analysis of strictly equivalent pairs would strengthen the conclusions.

  3. Final-answer evaluation provides limited insight into error mechanisms. Without systematic step-level analysis, the study cannot clearly distinguish failures in law selection, assumption interpretation, variable mapping, and subsequent computation.

  4. The DPO experiment does not establish a clear training conclusion. Differences between mathematics and physics training are statistically inconclusive, with limited evaluation size. The comparison between training subjects alone also does not establish improvement over the untrained baseline.

Questions for authors

Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.

10 comments
AuthorsOpenReview ↗

W1. Contribution, W2. Task Equivalence, and W3. Error Analysis (1/2)

W1. Contribution.

Our finding is not just that accuracy is lower in physics than in mathematics:

  • We measure the performance gap in a fair way. Why fair? Because each physics problem is paired with its mathematics source, and the two share a central derivation. We use average accuracy (single-trial accuracy) over 64 attempts to examine the gap systematically across reasoning settings (Section 4.3, Figure 6), models (Appendix B.5, Figure 19), mathematical difficulty, domain, authenticity, and physics-layer thickness (Section 4.4, Figure 7).
  • In Section 5.1 (Physics Capability Under Repeated Sampling), we then show that the gap is budget-dependent: if models have more time to think (more compute), the gap nearly disappears over 64 attempts (Figure 8).
  • That suggests a recipe: post-train on physics problems to make correct physics answers more likely on a single attempt. The revised Section 5.2 confirms it: post-training on the physics problems improves held-out accuracy in both physics (+4.11 pp) and mathematics (+3.49 pp), as much in mathematics as training on the mathematics problems themselves (see W4).
  • This raises the question: "We have many math datasets, but where do we find the physics problems?" Our answer is the skill we developed: physics experts converted math problems into physics problems, and we formalized this process and built a skill around it (in other words, we encoded the physicists' approach in a skill). We can now use this skill to create physics datasets.

We used the developed skill to make the comparison as fair as possible. The community can also use this skill to create physics datasets from existing mathematics datasets. To the best of our knowledge, this approach is novel and has never been used before. The released dataset and skill could be useful to the community.

W2. Task Equivalence.

We agree that, in principle, the mathematics and physics questions cannot be 100% identical in every respect. Our aim is to make the pairs as parallel as possible in the mathematics they require. We start from the source constraints and essential solution steps, then construct a physical system that encodes those relations. When the requested quantity changes, the correspondence explains how the physical observable relates to the mathematical target. We added more examples in Appendices A.3 and A.4 to show how the physics questions retain the mathematical structure and essential solution steps of their source problems.

W3. Step-Level Error Analysis.

A systematic analysis of chain-of-thought reasoning would require a dedicated study. Our contribution is to measure the performance gap between paired mathematics and physics problems and how it changes with the number of attempts. Final-answer evaluation supports these comparisons without attributing the gap to particular reasoning failures.

The revised Section 5.2 adds a response-level view of one failure mode. We automatically identify responses that exhaust the token budget and responses that end in repetitive loops (Appendix C.5). These account for 20.31% and 10.83% of the base model's physics responses, respectively, and post-training on physics reduces them to 15.52% and 8.13%; both reductions have 95% intervals that exclude zero.

AuthorsOpenReview ↗

W4. Training Conclusion and Improvement over the Untrained Baseline (2/2)

W4. Training Conclusion and Improvement over the Untrained Baseline.

Thank you for this suggestion. We extended Section 5.2 accordingly. The revised study compares each trained model with the untrained base model, Qwen3.5-4B, on both forms of the 60 held-out families. One model is trained on the physics form and one on the mathematics form of the same 156 training families: supervised fine-tuning on the base model's correct solutions, followed by preference optimization (IPO) on the model's own responses (Appendix C.2).

Training dataMathematics evaluationPhysics evaluation
None (base)64.6459.53
Mathematics68.70 (+4.06)60.31 (+0.78)
Physics68.13 (+3.49)63.65 (+4.11)
  • Improvement over the untrained baseline. Physics training raises physics accuracy by 4.11 pp (95% CI [1.20, 7.08]) and mathematics accuracy by 3.49 pp ([0.62, 6.41]), and mathematics training raises mathematics accuracy by 4.06 pp ([1.35, 6.93]). All three intervals exclude zero.
  • Evaluation size and robustness. Each model produces 3,840 scored responses (60 families, two subjects, 32 attempts). The intervals resample families within each collection and are paired across models. The physics-trained model exceeds the base model on all six physics collections and on five of the six mathematics collections (Table 19).
  • A clear training conclusion. Training on the physics problems improves both subjects. It matches mathematics training in mathematics (difference -0.57 pp, CI [-3.39, 2.14]) and exceeds it in physics by 3.33 pp (CI [1.09, 5.62]), so it also keeps the mathematics-physics gap 3.91 pp smaller than mathematics training does (CI [0.83, 7.19]). Its largest physics gains are on Hard problems, where accuracy rises from 7.81% to 12.89%, and it produces fewer budget-exhausting and looping responses, with intervals that exclude zero (Table 18).

The comparison of training subjects from the submitted version is kept in Appendix C.6. It points in the same direction as the new study: physics data gives higher physics accuracy than mathematics data.

AuthorsOpenReview ↗

Follow-up and Invitation to Discuss

Dear Reviewer 533A,

While we appreciate your recognition of our careful and auditable dataset construction, evaluation across multiple models and reasoning settings, with stronger support for the main findings, and paired design that helps control for differences in mathematical content, we would like to use the discussion period as an opportunity to improve our work further. We have made a good faith effort to address your concerns in our rebuttal and the revised version, and we hope that you will find our responses satisfactory. We believe our rebuttal addresses your concerns directly and reasonably well:

  • Contribution beyond the accuracy gap (W1): the connection between paired measurement, repeated sampling, and post-training, together with the reusable math-to-physics skill; addressed under "W1. Contribution" and in our general comments.
  • Task equivalence (W2): which constraints and essential solution steps the pairs retain, and how changes in the requested quantities relate to the mathematical targets; illustrated in Appendices A.3 and A.4.
  • Error analysis (W3): response-level analysis of budget exhaustion and repetitive loops (Appendix C.5), alongside the limits of final-answer evaluation for identifying step-level causes.
  • Training conclusion and the untrained baseline (W4): the revised study shows that physics post-training improves held-out accuracy in both physics (+4.11 pp) and mathematics (+3.49 pp) over the baseline (Section 5.2, Table 2).

We would appreciate it if you could confirm whether these responses address your concerns. If any point remains unresolved, please identify it specifically so that we can respond more precisely or acknowledge a limitation that cannot be resolved during the discussion period. We invite you to continue the conversation.

Best regards,

The authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revised Version and Invitation to Discussion

Dear Reviewers,

We have submitted a revised version of the paper to address your comments.

We would like to summarize our contributions and invite you to join the discussion.

The central question of the paper is how to measure the performance gap between math and physics in a fair way. Our answer to the question "Can LLMs keep up when math becomes physics?" is:

  • No, on average, they can't. (Single-trial accuracy is lower in physics than in mathematics.)
  • Yes, if we give LLMs more time to think (so the gap is budget-dependent, and it nearly disappears with 64 attempts.)

The takeaway is that post-training on physics can improve LLM performance in both physics and math. The revised Section 5.2 shows this on held-out problems (see our comment "New Results: Post-Training on Physics Improves Both Physics and Mathematics"). This is also an invitation to the community to consider using physics in post-training instead of relying heavily on math. There are fewer physics datasets than math datasets, but we have developed a skill for converting math problems into physics problems by learning from physics experts, and we have released a dataset of 216 physics problems. The community can use this skill to create more physics datasets from existing math datasets.

To summarize:

  • We introduce Math2Physics: a dataset of 216 expert-reviewed mathematics-physics pairs and a conversion skill developed from expert-written examples (Sections 3.1 and 3.2).
  • We measure the physics-math performance gap in a fair way, since each physics problem shares a central derivation with its mathematics source (Sections 3.1 and 4.1).
  • This measurement is based on:
    • Single-attempt accuracy estimated from 64 attempts per question across eight reasoning settings (Section 4.1; Table 1).
    • Comparisons by reasoning effort (Section 4.3, Figure 6), mathematical difficulty (Section 4.4, Figure 7a), physical domain (Section 4.4, Figure 7b), authenticity (Section 4.4, Figure 7c), and physics-layer thickness (Section 4.4, Figure 7d).
    • Correlations between models' problem-level gaps (Appendix B.5, Figure 19).
    • Oracle pass@k over 64 attempts, which shows that the gap nearly disappears (Section 5.1) and motivates post-training on physics.
  • We show that post-training on the physics problems improves held-out accuracy in both physics and mathematics, with the largest physics gains on hard problems, and narrows the mathematics-physics gap compared with training on mathematics (Section 5.2, Table 2; Appendix C.2-C.5).
AuthorsOpenReview ↗

Post-Training on Physics Improves Both Physics and Mathematics

Dear Reviewers,

All three reviews asked what the training experiment in Section 5.2 establishes. Reviewer 533A asked for a comparison with the untrained baseline, Reviewer uUNv for a clearer conclusion, and Reviewer tLYx for its connection to the rest of the paper. We have extended Section 5.2 into a full post-training study, with all details in Appendix C.2-C.5.

What we did

  • Starting from Qwen3.5-4B, we train one model on the physics form and one on the mathematics form of the same 156 training families. The remaining 60 families (ten per collection) are held out.
  • Each model learns only from its own subject, in two stages: supervised fine-tuning on the base model's correct solutions, then preference optimization (IPO) on the model's own full-length responses. The preference step favors correct answers over incorrect ones and over responses that loop or exhaust the token budget.
  • The base model and both trained models answer both the mathematics and the physics form of every held-out family: 32 attempts each, an 81,920-token budget, and strict scoring.

Results (Table 2, held-out accuracy in %, gain over the base model in parentheses)

Training dataMathematics evaluationPhysics evaluation
None (base)64.6459.53
Mathematics68.70 (+4.06)60.31 (+0.78)
Physics68.13 (+3.49)63.65 (+4.11)
  • Physics training improves both subjects, with statistically clear gains. It raises physics accuracy by 4.11 pp [1.20, 7.08] and mathematics accuracy by 3.49 pp [0.62, 6.41]; both 95% paired-bootstrap intervals exclude zero. The physics-trained model exceeds the base model on all six physics collections and on five of the six mathematics collections (Table 19).
  • Physics data is the more valuable training data. In mathematics, physics training matches mathematics training (difference -0.57 pp, CI [-3.39, 2.14]); in physics, it exceeds mathematics training by 3.33 pp (CI [1.09, 5.62]). Mathematics training raises mathematics by 4.06 pp [1.35, 6.93] but physics by only 0.78 pp.
  • Physics training narrows the mathematics-physics gap while still improving mathematics. Mathematics training widens the base model's gap from 5.10 to 8.39 pp, whereas physics training narrows it to 4.48 pp, 3.91 pp less (CI [0.83, 7.19]). Adding converted physics problems to mathematics-heavy post-training is therefore a promising way to reduce the gap measured in Section 4.
  • Gains on hard problems. On Hard physics problems (Section 3.4), physics training raises accuracy from 7.81% to 12.89% (+5.08 pp, CI [0.78, 9.77]) and solves 2 of the 7 held-out physics problems that the base model never solves in 32 attempts. In mathematics, the largest gains are on Medium problems (+10.74 pp with mathematics training, +7.03 pp with physics training).
  • Coverage, consensus, and reliability improve. Physics training raises pass@32 from 88.33% to 91.67% in physics and from 93.33% to 96.67% in mathematics, and majority-vote accuracy from 61.67% to 65.00% and from 66.67% to 75.00%. Budget-exhausting responses fall from 20.31% to 15.52% in physics and from 19.53% to 13.39% in mathematics, and repetitive loops from 10.83% to 8.13% and from 11.72% to 8.18%; all four reductions have 95% intervals that exclude zero. Mean response length falls by 12-17% (Table 18).

How this completes the paper's story

  1. Section 4 measures the gap fairly: on paired problems, physics single-attempt accuracy is 5.71 pp lower.
  2. Section 5.1 shows that the gap concerns reliability: with 64 attempts, oracle coverage is 93.11% for physics and 93.90% for mathematics.
  3. Section 5.2 shows that post-training on the physics problems turns this coverage into single-attempt accuracy, in physics and in mathematics, and narrows the gap compared with mathematics training.

Because our skill converts existing mathematics collections into physics problems, Math2Physics provides a scalable source of post-training data that benefits both subjects. The comparison of training subjects from the submitted version is kept in Appendix C.6; it points in the same direction, with physics data giving higher physics accuracy than mathematics data.

We updated the abstract, introduction, contributions, and conclusion to reflect these results, using the additional page for Section 5.2.

AuthorsOpenReview ↗

What we did.

Dear Reviewers and AC,

This summary brings together the main points from our submission and the discussion. We invite the reviewers to continue the discussion.

  1. A fair comparison of LLM performance on math and physics problems has not been done before. This is the main motivation behind our work.

  2. We show that when math becomes physics, LLMs can't keep up in average accuracy (single-run accuracy averaged over 64 attempts, Table 1), but they can keep up if they have more time to think (i.e., more compute budget). This goes beyond the common observation that more compute can improve performance: Physics gains more in oracle coverage than math, and the gap nearly disappears (Figure 8). This motivates post-training on physics.

  3. To answer "Can LLMs keep up?", we compare the paired mathematics-physics gap across reasoning efforts (Section 4.3, Figure 6), test whether the gap happens on the same problem pairs across models (Appendix B.5, Figure 19), and report subgroup gaps by mathematics difficulty, physical domain (e.g., Classical Mechanics, Electromagnetism), authenticity (i.e., how naturally the scenario represents a physical process or measurement task), and physics-layer thickness (i.e., the number of added physical interpretation steps) (Figure 7).

  4. Our experts (human physicists) develop a framework for converting math problems into physics problems. They fuse the mathematical derivation with a physical context (layer) to create the parallel physics problem (Section 3.1). This forms the basis of our Math2Physics dataset.

  5. Further, by separating physical abstractions from the underlying derivations in the base dataset, we develop an LLM skill for converting more math problems into physics problems (Section 3.2).

  6. We release the Math2Physics dataset of 216 expert-reviewed math-physics pairs, which can be used to train LLMs on physics. The community can create more physics datasets from existing math datasets by applying the Math2Physics skill (Appendix D.2).

  7. Post-training on physics improves accuracy in both physics and mathematics and yields a smaller gap than training on the paired mathematics problems (Section 5.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.