- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TIRE is a verifiable data synthesis framework for physics reasoning. It first extracts formulas and applicability conditions from existing physics problems to build a reusable knowledge base, then generates computational instances through program synthesis, unit testing, and parameter instantiation, using program execution results as reference answers to filter teacher-model reasoning trajectories for SFT and GRPO training. The authors build TIRE-19k, a dataset of 19,144 examples across nine physics subfields. On three backbone models and four benchmarks, it improves average accuracy over the strongest baseline by 2.86, 2.15, and 3.57 percentage points, respectively; ablations support unit testing and parameter perturbation, and GRPO further improves Qwen3-8B by 1.52 percentage points over SFT.
Critical strengths
-
Grounding problem construction in physics knowledge and verifying it through execution yields more reliable numerical supervision. Formulas and applicability conditions feed the generation pipeline; program synthesis, unit testing, and parameter instantiation produce executable computations whose results verify teacher-model answers. This reduces numerical errors from model-generated answers and offers a practical path to scaling physics reasoning data.
-
Experiments support the data's practical value. With training examples held constant, TIRE-19k achieves the highest average accuracy across four benchmarks on all three backbones. Ablations confirm the benefits of unit testing and parameter perturbation, and PHYSICS-only results show the advantage persists after removing data taken directly from evaluation benchmarks, strengthening confidence in the conclusions.
Critical weaknesses
- Insufficient direct evaluation of the correctness of the final synthetic data. The mechanisms used in the paper cannot independently establish whether the generated scenarios satisfy the applicability conditions of the formulas, nor do they verify the teacher model's reasoning steps. The paper does not provide an independent sample-based audit to quantify the physical correctness of the retained problems, the correctness of the reasoning steps, or the extent to which each verification stage actually reduces errors. Thus, the current results support the downstream utility of the data. However, given that reliable data synthesis is the central research objective, this omission limits the empirical support for the core contribution.
- Limited methodological novelty and insufficient differentiation from existing approaches. The proposed framework mainly integrates established techniques, including formula extraction, program synthesis, execution-based verification, and teacher distillation. Compared with existing verifiable data synthesis methods, TIRE primarily distinguishes itself by incorporating physics-specific applicability conditions, which are largely enforced through prompting and LLM-based judgments rather than a novel technical mechanism. Moreover, the ablation studies do not isolate the contribution of these physics-specific constraints. Consequently, the work appears to be primarily a systematic integration and domain adaptation of existing techniques, with limited methodological novelty.
Questions for authors
-
Direct evidence of data correctness Section 3.4 explicitly states that verification checks only the consistency of the final numerical answers and does not verify intermediate reasoning steps. Have the authors conducted an independent sample-based evaluation of the retained data to examine whether the formulas' applicability conditions are met, whether the generated scenarios are physically plausible, and whether the teacher model's reasoning steps are correct? If so, please describe the evaluation method, sample size, coverage of data sources, and error rates. Such evidence would directly affect my assessment of the reliability of the data.
-
Training budget and attribution of performance gains TIRE and the strongest baseline, SCP-116k, use the same maximum sequence length and thinking mode, but equal numbers of examples may still entail different numbers of training tokens and computational budgets. Is there evidence that disentangles the contribution of the generation mechanism from the effect of the length of reasoning supervision? The answer would affect my assessment of whether the performance advantage stems primarily from TIRE's data construction mechanism.