- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the performance of current LLMs on math problems and their equivalent physical problems. It introduces Math2Physics, a benchmark containing 216 problem pairs, and a math-to-physics skill for converting math problems into physics problems. Experiments show that models generally perform worse on physics problems than on their mathematical counterparts, but this gap narrows substantially as the number of sampling attempts increases.
Critical strengths
-
The paper reveals a performance gap between current LLMs’ ability to solve physics problems and their ability to solve equivalent mathematical problems.
-
It proposes a math-to-physics skill that may be useful for the community to synthesize physics data.
Critical weaknesses
-
The Math2Physics benchmark is too small and too easy to clearly distinguish the capabilities of frontier LLMs.
-
The paper lacks an analysis of the performance gap. It only reports the existence of the gap without explaining its causes.
-
The gap narrows to 0.6% with 64 attempts, suggesting that it may arise from sampling randomness rather than a systematic capability gap. This limits the significance of the this research.
-
The purpose of the DPO experiment is unclear. Post-training LLMs on physics data would naturally be expected to improve their performance, but this experiment does not seem to contribute to the paper’s overall analysis or conclusions.
Questions for authors
I am curious about the detailed implementation of the math-to-physics skill. How does it transform an abstract mathematical problem into concrete physics problems across different domains?
9 comments
W3. Repeated Sampling, and Question: Math-to-Physics Skill (2/3)
W3. The gap narrows to 0.6% with 64 attempts, suggesting that it may arise from sampling randomness rather than a systematic capability gap. This limits the significance of this research.
This comment seems to conflate two measures: single-attempt accuracy and oracle pass@k.
-
Single-attempt accuracy (Section 4.2, Table 1). We estimate how often an individual answer is correct using 64 attempts per question; this is simply average accuracy. The median of mathematics-physics gap is is 6.27 percentage points. So yes, the gap is systematic as Table 1 shows.
-
Oracle pass@64 (Section 5.1, Figure 8). This counts a problem as solved if any of 64 answers is correct. We further explain what narrowing the gap shows in our response to W4 (3/3).
This is actually not a weakness rather it is one of the strong observations of the paper:
LLM physics performance is lower in single-trial accuracy, but it becomes close to math if models have more time/compute.
The revised Section 5.2 builds on this observation: post-training on the physics problems turns this coverage into single-attempt accuracy on held-out problems, by 4.11 pp in physics and 3.49 pp in mathematics.
Questions
I am curious about the detailed implementation of the math-to-physics skill. How does it transform an abstract mathematical problem into concrete physics problems across different domains?
Physics experts first converted 30 mathematics problems into physics counterparts while preserving the underlying mathematics. We distinguish two levels: a physical model of a phenomenon and the mathematical abstraction of that model. Starting from the mathematical abstraction, the experts construct a physical model that realizes it. We call this step "expert construction" (Figure 2). We then encoded the conversion procedure illustrated by the 30 CSAT mathematics-physics pairs as a set of instructions for LLMs.
For a new mathematics problem, an agent first solves it and records its variables, constraints, requested quantity, and essential solution steps. It then proposes three candidate physical models and selects one whose laws and measurements recover the same mathematical relationships. For example, optical reflection and travel times recover geometric constraints (Figure 9), while a volume integral becomes a moment-of-inertia calculation under a specified density (Figure 10). The agent then writes and solves the physics problem and checks the mathematics, physical laws, units, alternative interpretations, and shortcuts. The construction is retained if all checks pass.
In response to your question, we added an example of a skill-guided conversion in Section 3.2 (Figure 3) and four further examples in Appendix A.4. Please let us know if you have any further questions about the skill.
W4. Purpose of the Training Experiment (3/3)
W4. The purpose of the DPO experiment is unclear. Post-training LLMs on physics data would naturally be expected to improve their performance, but this experiment does not seem to contribute to the paper's overall analysis or conclusions.
We revised Section 5.2 to make its purpose and contribution explicit. The experiment tests the paper's main takeaway: physics counterparts of mathematics problems are a useful source of post-training data. Section 5.1 shows that correct physics answers are available for most problems (oracle coverage 93.11% at 64 attempts) but are produced less reliably on a single attempt. Section 5.2 asks whether post-training turns this coverage into single-attempt accuracy, and how physics data compares with mathematics data.
The revised study trains Qwen3.5-4B on either the physics or the mathematics form of the same 156 families and evaluates both forms of 60 held-out families against the untrained base model (Table 2). The results go beyond "physics data improves physics":
- Physics training improves both subjects: +4.11 pp in physics (95% CI [1.20, 7.08]) and +3.49 pp in mathematics ([0.62, 6.41]). Mathematics training improves mathematics (+4.06 pp) but adds only 0.78 pp in physics. The transfer to mathematics is consistent with the physics problems retaining the mathematical structure of their sources, which only the paired design can test: both models learn from the same families and differ only in formulation.
- Physics training narrows the gap the paper measures: mathematics training widens the base model's mathematics-physics gap from 5.10 to 8.39 pp, whereas physics training narrows it to 4.48 pp (3.91 pp less; CI [0.83, 7.19]) while improving mathematics as much as mathematics training does.
- Gains on hard problems: on Hard physics problems, accuracy rises from 7.81% to 12.89%, and the physics-trained model solves 2 of the 7 held-out physics problems that the base model never solves in 32 attempts.
- Responses become more reliable: budget-exhausting responses fall from 20.31% to 15.52%, repetitive loops from 10.83% to 8.13%, and responses become shorter.
These results connect the three parts of the paper: a fair measurement of the gap (Section 4), evidence that the gap concerns reliability rather than the availability of correct solutions (Section 5.1), and a demonstration that physics problems built with our skill improve single-attempt accuracy in both subjects and narrow the gap (Section 5.2). Per-collection results are in Table 19, and further analyses in Appendix C.5.
W1. Benchmark Size and Difficulty, and W2. Gap Analysis (1/3)
W1. The Math2Physics benchmark is too small and too easy to clearly distinguish the capabilities of frontier LLMs.
Could you clarify how many problems you would consider sufficient and what level of difficulty you would not consider "too" easy?
Math2Physics contains 216 expert-written/reviewed mathematics-physics pairs. The mathematical datasets we use are common in the community; for example, MathArena reports accuracy for Qwen3-235B-A22B, DeepSeek-R1-0528, GLM 4.5, Kimi K2, gpt-oss, and many other models on SMT 2025 and CMIMC 2025 [1]. Almost all model cards report performance on AIME. Harder datasets include, for example, IMO-ProofBench (Olympiad-level proof problems) [2], Riemann-Bench (25 research-level problems) [3], and HorizonMath (for mathematical discovery) [4]. These target specialized tasks in proof writing and research mathematics. Competition-math benchmarks such as AIME and HMMT recur across model-release reports and academic evaluations [5-9].
These harder benchmarks have specialized applications in developing Olympiad proof-writing systems and assessing progress toward mathematical discovery. These applications are outside the scope of our paper.
Therefore, we believe our choice of mathematical datasets follows common LLM evaluation practice in both size and difficulty.
In addition, using the skill we developed, anyone in the community can create their own physics datasets from existing mathematics datasets (Appendix D.2). We encourage the reviewer to read Appendix A, which shows how existing mathematics problems can be converted into physics problems while preserving the same logical structure. This transformation is a reusable framework that allows us to further expand the Math2Physics analysis by following the skill guidelines.
If the base dataset (the math part) is difficult, the resulting physics problems tend to be difficult too (Appendix B.7, Figure 20a). We also encourage the reviewer to look at the examples in Figure 3 and Appendix A.3 (Figures 9 and 12); these are not easy problems. We also refer the reviewer to our difficulty analysis in Section 3.4 and Appendix B.11 (Table 16).
W2. The paper lacks an analysis of the performance gap. It only reports the existence of the gap without explaining its causes.
We respectfully disagree. Here is why:
We devote
Section 4. The Math-Physics Performance Gapto analyzing the gap. We compare eight model and reasoning settings, including no thinking and different levels of reasoning effort (low, medium, and (x)high). We include both dense and MoE models to examine how performance changes across settings and models.Reasoning effort (Section 4.3, Figure 6; Appendix B.4, Tables 7-8). More reasoning does not consistently close the gap.
Recurring problems across models (Section 4.4; Appendix B.5, Figure 19). Different models tend to show the gap on the same problems (correlations: 0.282-0.485).
Mathematical difficulty (Section 4.4, Figure 7a; Appendix B.6, Table 9; Appendix B.7, Figure 20a and Table 11). The gap is 7.12 pp for Easy problems, 3.77 pp for Medium, and -0.95 pp for Hard.
Physical characteristics (Section 4.4, Figures 7b-d; Appendix B.6, Tables 9-10). We examine the physics topic, how natural the scenario is, and the number of added physics interpretation steps.
Repeated sampling (Test-time scaling) (Section 5.1, Figure 8; Appendix B.8, Figure 20b; Appendix C.1, Figure 21). When a problem counts as solved if any attempt is correct, the gap falls from 5.71 pp with one attempt to 0.80 pp with 64 attempts.
Scoring and evaluation choices (Section 4.2; Appendix B.9, Table 12). The gap remains 5.71 pp when answers may omit units and is 10.02 pp under strict scoring.
Performance across collections and settings (Section 4.2, Figure 5; Appendix B.3, Figure 18 and Tables 5-6). Leaving out any one collection leaves an overall gap of 4.99-6.72 pp.
References
[1] MathArena Team. MathArena, 2025.
[2] Luong et al. Towards Robust Mathematical Reasoning. EMNLP 2025.
[3] Garre et al. Riemann-Bench: A Benchmark for Moonshot Mathematics. 2026.
[4] Wang et al. Measuring Progress in Reasoning Toward Mathematical Discovery with Automatic Verification. 2026.
[5] Dekoninck et al. Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs. 2026.
[6] Qwen Team. Qwen3.6-35B-A3B. Official model card. 2026.
[7] Meta. Muse Glimmer-30B: Model Card and Evaluation Methodology. 2026.
[8] Microsoft. Phi-4-Reasoning Technical Report. 2025.
[9] NVIDIA. Nemotron-Cascade-2. Technical report. 2026.