acceptodds
Under review as a conference paper at ICLR 2027

TIRE: Top-Down Formula Instantiation with Reliable Execution for Physics Problem Synthesis

Abstract

Advancing physics reasoning in large language models requires scalable training data that connect physical laws, concrete problem scenarios, and verifiable numerical answers. Expert-designed problems offer top-down control over physical laws, conditions, and scenarios, but require costly human effort. LLM-based synthesis from texts scales efficiently, but model-generated answers and assessments leave supervision vulnerable to hallucinations. We introduce **TIRE**, a framework for Top-Down Formula Instantiation with Reliable Execution that brings knowledge-driven problem construction to large-scale data generation. Top-down instantiation uses reusable physical formulas and their applicability conditions to control the physical relations and numerical configurations of generated problems. Reliable execution establishes reference answers before problem generation, providing a common numerical foundation for problem construction and teacher distillation. Together, these mechanisms combine control over problem content with execution-grounded supervision for both supervised fine-tuning and reinforcement learning. Using TIRE, we construct **TIRE-19k**, spanning nine subfields of physics. Across **3** backbone LLMs and **4** benchmarks, TIRE-19k outperforms **5** representative datasets with equal amounts of SFT data, exceeding the strongest baseline by **2.86 percentage points** on average. Further GRPO raises Qwen3-8B's average accuracy by **1.52 percentage points** over its TIRE-19k SFT checkpoint. These results demonstrate the value of generating physics training data from explicit physical knowledge with reliable execution.

Primary area:
datasets and benchmarks
Keywords:
Physics Reasoning, Synthetic Data Generation, Program-Guided Synthesis, Execution-Based Verification, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FipJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TIRE is a verifiable data synthesis framework for physics reasoning. It first extracts formulas and applicability conditions from existing physics problems to build a reusable knowledge base, then generates computational instances through program synthesis, unit testing, and parameter instantiation, using program execution results as reference answers to filter teacher-model reasoning trajectories for SFT and GRPO training. The authors build TIRE-19k, a dataset of 19,144 examples across nine physics subfields. On three backbone models and four benchmarks, it improves average accuracy over the strongest baseline by 2.86, 2.15, and 3.57 percentage points, respectively; ablations support unit testing and parameter perturbation, and GRPO further improves Qwen3-8B by 1.52 percentage points over SFT.

Critical strengths
  • Grounding problem construction in physics knowledge and verifying it through execution yields more reliable numerical supervision. Formulas and applicability conditions feed the generation pipeline; program synthesis, unit testing, and parameter instantiation produce executable computations whose results verify teacher-model answers. This reduces numerical errors from model-generated answers and offers a practical path to scaling physics reasoning data.

  • Experiments support the data's practical value. With training examples held constant, TIRE-19k achieves the highest average accuracy across four benchmarks on all three backbones. Ablations confirm the benefits of unit testing and parameter perturbation, and PHYSICS-only results show the advantage persists after removing data taken directly from evaluation benchmarks, strengthening confidence in the conclusions.

Critical weaknesses
  • Insufficient direct evaluation of the correctness of the final synthetic data. The mechanisms used in the paper cannot independently establish whether the generated scenarios satisfy the applicability conditions of the formulas, nor do they verify the teacher model's reasoning steps. The paper does not provide an independent sample-based audit to quantify the physical correctness of the retained problems, the correctness of the reasoning steps, or the extent to which each verification stage actually reduces errors. Thus, the current results support the downstream utility of the data. However, given that reliable data synthesis is the central research objective, this omission limits the empirical support for the core contribution.
  • Limited methodological novelty and insufficient differentiation from existing approaches. The proposed framework mainly integrates established techniques, including formula extraction, program synthesis, execution-based verification, and teacher distillation. Compared with existing verifiable data synthesis methods, TIRE primarily distinguishes itself by incorporating physics-specific applicability conditions, which are largely enforced through prompting and LLM-based judgments rather than a novel technical mechanism. Moreover, the ablation studies do not isolate the contribution of these physics-specific constraints. Consequently, the work appears to be primarily a systematic integration and domain adaptation of existing techniques, with limited methodological novelty.
Questions for authors
  • Direct evidence of data correctness Section 3.4 explicitly states that verification checks only the consistency of the final numerical answers and does not verify intermediate reasoning steps. Have the authors conducted an independent sample-based evaluation of the retained data to examine whether the formulas' applicability conditions are met, whether the generated scenarios are physically plausible, and whether the teacher model's reasoning steps are correct? If so, please describe the evaluation method, sample size, coverage of data sources, and error rates. Such evidence would directly affect my assessment of the reliability of the data.

  • Training budget and attribution of performance gains TIRE and the strongest baseline, SCP-116k, use the same maximum sequence length and thinking mode, but equal numbers of examples may still entail different numbers of training tokens and computational budgets. Is there evidence that disentangles the contribution of the generation mechanism from the effect of the length of reasoning supervision? The answer would affect my assessment of whether the performance advantage stems primarily from TIRE's data construction mechanism.

Reviewer_SaufOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIRE, a framework for physics reasoning data synthesis based on top-down formula instantiation and reliable execution. Starting from physical formulas and their applicability conditions, the framework constructs problems through program generation and unit testing, parameter instantiation, and numerical execution. Execution-derived results are then used to verify the final answers produced by a teacher model, generating training data with reasoning traces. The authors construct TIRE-19k, a dataset containing 19,144 examples across nine subfields of physics. Experiments with three backbone models on four benchmarks show that TIRE-19k generally outperforms five baseline datasets, while further GRPO training provides additional performance gains.

Critical strengths
  1. The paper uses physical formulas and their applicability conditions as the starting point for data synthesis, rather than relying directly on language models to generate complete question-answer pairs. By constructing numerical instances before generating natural-language problems, the framework provides more explicit control over physical relations and parameter configurations, making its design well suited to physics reasoning tasks.
  2. The framework integrates program validation, instance execution, and teacher-answer verification. Reference answers are established before problem generation and teacher inference, and the teacher model does not receive these answers directly. This helps reduce numerical errors that may arise when relying entirely on language models for self-evaluation.
  3. The experiments cover three backbone models, four benchmarks, and five baseline datasets, along with component ablations, teacher sensitivity analysis, PHYSICS-only training, and GRPO experiments. The generally consistent results suggest that the constructed dataset has a degree of cross-model applicability and generalization potential.
Critical weaknesses
  1. Reliable execution primarily ensures the computational consistency of formula-based programs, but does not necessarily guarantee the physical correctness of generated problems. The formulas themselves undergo LLM-based plausibility filtering, unit-test expectations are derived from the same formulas, and teacher verification only checks whether final answers fall within a 5% tolerance. Consequently, incorrect physical assumptions, inappropriate formula applications, and erroneous intermediate reasoning may still pass verification. The paper lacks an independent evaluation of physical and semantic correctness.
  2. Although the main experiments use equal numbers of training examples, the training configurations differ across datasets. Table 10 in the appendix shows that TIRE uses a maximum sequence length of 40,960, whereas some baselines use only 8,192 or 16,384, and thinking mode is disabled for MegaScience and Spark. These differences make it difficult to directly compare training token counts, reasoning supervision formats, and computational budgets. The current results therefore do not fully disentangle the contributions of the data construction method from those of the training configurations.
  3. The component ablations provide limited evidence for the independent contributions of individual components. Appendix B.2 indicates that the w/o unit tests variant contains only 2,256 verified examples from programs that failed unit tests, with the remaining examples resampled from the full TIRE-19k dataset. Meanwhile, the w/o all variant simultaneously removes program synthesis, parameter generation, and execution. Consequently, the performance differences in Table 5 do not sufficiently isolate the individual contributions of unit testing, reliable execution, and other generation stages.
Questions for authors

Please refer to Weaknesses.

Reviewer_56wzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TIRE, an automated pipeline that synthesizes physics reasoning data "top-down" from formula abstractions. The framework operates in six stages: (1) extracting formula records =(name,equation,description,applicability) from textbook and qualifier problems (PHYSICS and SciBench) using DeepSeek-R1; (2) generating Python solvers and unit tests with DeepSeek-V3.1, retaining only code passing all tests; (3) proposing parameter values (with ±5%\pm 5\% perturbation on SciBench items) and executing scripts to establish a numeric reference y*; (4) verbalizing the record into natural language; (5) generating long Chain-of-Thought (CoT) solutions using Qwen3-235B-A22B-Thinking (K=1); and (6) filtering traces to retain only those within 5%5\% relative error of y*.

The resulting dataset, TIRE-19k (19,144 problems across 3,051 formulas), is evaluated via SFT across three backbones (Qwen3-8B, DeepSeek-R1-Distill-Qwen-7B, Qwen3.5-9B) against five matched-size (19,144) public physics/science datasets across four benchmarks (SciBench, PhysReason, OlympiadBench, TheoremQA). TIRE-19k achieves the highest four-benchmark average across all backbones (+2.15 to +3.57 points). A PHYSICS-only ablation isolates SciBench contamination, and GRPO on top of SFT provides an additional +1.52 average gain on Qwen3-8B.

Critical strengths

The methodology is thoroughly detailed, including complete prompts for all nine LLM stages, exact decoding parameters, benchmark subset definitions, failure-handling semantics, and extraction rules (Appendices A–C). Training on a PHYSICS-only variant (Table 8) completely removes SciBench-derived formulas while maintaining a +3.12 point advantage over the strongest baseline, effectively mitigating the immediate concern of SciBench evaluation contamination. The w/o all (Self-QA) ablation in Table 5 holds the Qwen3-235B teacher fixed while removing program execution, demonstrating a clear drop (-3.76 overall, -9.33 on OlympiadBench). Directional improvements over existing datasets (PhysLogic, Spark, SCP-116k) replicate across three distinct model families and sizes. Truncated, malformed, or unparseable outputs are strictly scored as incorrect without manual lenient extraction, and all reported table deltas and averages are arithmetically consistent.

Critical weaknesses

Severe Experimental Confounds in Baseline Comparisons: The central claim that formula-instantiated, execution-verified problems drive performance gains (L368–370) is confounded by four major training discrepancies:

TIRE solutions are generated by a frontier 2025 reasoning model (Qwen3-235B-A22B-Thinking) with long traces (mean 5,136 tokens; 22% over 8k tokens). Baselines utilize original, shorter, or older teacher traces (e.g., DeepSeek-R1/o1-mini). Two baselines (MegaScience and Spark) are fine-tuned and evaluated with thinking disabled (L798–800), while TIRE and base backbones run with thinking enabled, skewing baseline scores downward. Maximum context length varies dramatically across datasets (8,192 for MegaScience/PhysLogic vs. 40,960 for TIRE/SCP). The paper lacks a same-teacher re-distillation baseline (i.e., re-solving SCP-116k or OpenSci physics questions using Qwen3-235B under the same length and reference filtering). Without this, the gain cannot be separated from the distillation of long-CoT reasoning traces. Omission of Untuned Base Model Baselines: Because all three backbones are native reasoning models, omitting the zero-shot / untuned backbone scores makes it impossible to determine whether SFT on TIRE-19k genuinely enhances physics reasoning capabilities or simply induces less degradation than noisier external corpora. Statistical Fragility on Low-Count Test Sets: All results are single-run evaluations using 1 sampled response (T=0.6/1.0) on small benchmarks: TheoremQA has 114 questions (1 question = 0.88%) and OlympiadBench has 236 questions (1 question = 0.42%). Crucially, altering training hyperparameters between Table 2 and Table 5 (reducing max length from 40k to 24k and adjusting batch size) shifts TIRE scores by -2.47 on SciBench and -1.69 on OlympiadBench. This configuration-induced variance matches or exceeds the ablation deltas reported in Table 5 (-1.38 to -1.43).

Confounded Component Ablation Designs

Unit Tests: The w/o unit tests ablation replaces 2,256 examples with items from failing programs that nevertheless passed teacher agreement. Because the teacher verified the outputs, many of these references are likely correct while the unit tests were faulty; swapping 12% of the data does not isolate unit-test efficacy. Perturbation: Parameter perturbation is applied exclusively to SciBench-derived items (21% of data), leaving the majority of the dataset unperturbed without justification. Self-QA: The w/o all condition simultaneously alters the verbalizer model, reference mechanism, and selection criteria, confounding the attribution to execution verification.

Format Leakage and Evaluation Formatting Alignment: Formulas are extracted directly from the 193 SciBench evaluation problems and official solutions. Furthermore, the teacher generation prompt appends "The unit of the answer is {u}" and enforces boxed unitless answers (L1237–1241), identical to the SciBench evaluation prompt. While Table 8 shows gains survive on PHYSICS-only, a systematic n-gram and embedding a decontamination audit against all four benchmarks is missing.

Unverified Execution Correctness and K=1 Filtering Bias: No human or expert audit is provided to verify the physical validity of generated formulas, applicability parameters, or final text problems. Generating code and tests via the same model risks correlated failure modes. Furthermore, requiring the teacher to solve problems on the first try (K=1) introduces a survivorship bias toward simpler problems within the teacher's immediate capability. Overstated GRPO and Teacher Generalization Claims: GRPO is applied only against the two weakest baselines (MegaScience and PhysLogic), omitting competitive sets like SCP-116k. The text highlights out-of-domain improvements while omitting that TIRE’s accuracy declines on PhysReason after GRPO (78.19 to 77.63) and gains less on SciBench (+0.56) than the baselines (+1.68 and +1.37).

Minor Issues Missing Closely Related Physics Literature (W8): Omits Infinite Problem Generator (Sharan et al., arXiv 2603.14486, Mar 2026; "Formula-as-Code" execution verification) and SymPyBench (arXiv 2512.05954; parameterized physics code execution). Absence of Scaling Curves and Cost Accounting (W9): Claims "scalable" data generation without a scaling curve (e.g., 5k, 10k, 19k) or compute/token cost reporting. Sample Count Discrepancy (W12): L247 and L335 state 19,144 training examples, while Table 10 indicates a 2% holdout leaving 18,762 examples. Table 8 reaches 19,144 on PHYSICS by duplicating ≈21%\approx 21\% of its 15,133 examples.

The paper provides a clear, exceptionally well-documented synthesis pipeline and strong engineering discipline, supported by a compelling PHYSICS-only decontamination experiment (Table 8). However, the primary empirical claims confound data synthesis methodology with teacher reasoning strength (long-CoT Qwen3-235B vs. older/disabled-thinking baselines), context window length, and answer formatting. Evaluating single-sample outputs on small test sets yields performance differences that fall within the variance observed across minor training configuration adjustments. Addressing the same-teacher re-distillation baseline and establishing multi-seed statistical significance would bring this work above the acceptance threshold.

Questions for authors

Can you report performance when the physics questions from SCP-116k or OpenSci are re-distilled through Qwen3-235B-A22B-Thinking at matched max context length (40,960) and 5%5\% reference verification? What are the zero-shot/untuned scores for Qwen3-8B, DeepSeek-R1-Distill-Qwen-7B, and Qwen3.5-9B under your exact evaluation harness? Why does TIRE's performance on SciBench drop by 2.47 points between Table 2 (72.27 avg) and Table 5 (71.51 avg) on the same dataset? How do multi-seed runs (3 seeds) or avg@k sampling affect the statistical significance of the 1.4-point ablation differences? What percentage of LLM-generated Python programs pass unit tests, what fraction fail execution, and what is the teacher-agreement rate per source? In a sample of retained vs. rejected problems (e.g., n=100), what fraction contains physically well-posed questions with exact numeric answers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.