- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the influence of structured physical scene descriptions on LLM physics reasoning. The authors introduce a six-field Scene Contract and evaluate matched, mismatched, and single-field perturbed contracts across three physics benchmarks and four models. The paper also examines whether inline self-checking improves final-answer accuracy.
Critical strengths
- The paper highlights physical scene modeling as an important component of physics problem-solving.
- The matched, mismatched, and field-level perturbation experiments provide a structured approach to evaluating sensitivity to physical descriptions.
- The evaluation covers multiple benchmarks and models, with statistical analyses supporting the observed performance differences.
Critical weaknesses
- Identifying physical states, constraints, and stage transitions often requires substantial reasoning and computation, constituting a core challenge in complex physics problems. Providing a complete, correct six-field scene contract may bypass these essential reasoning steps, raising concerns about the validity of this experimental setup.
- The matched scene contracts are constructed using reference solutions, potentially providing privileged modeling information that simplifies the original tasks. The reported gains may therefore overstate the contribution of scene information alone.
- Replacing an entire contract with one from another problem may introduce severe contradictions, while relevant and irrelevant field edits may differ in semantic severity. The observed performance drops therefore do not necessarily demonstrate a distinctive physical scene reasoning mechanism.
- Only final-answer accuracy is evaluated, without examining intermediate physical modeling decisions. Additionally, the inline self-checking comparison lacks matched-run and compute-controlled baselines.
Questions for authors
- How do the authors justify treating the six-field contract as a pre-derivation scene representation when identifying physical states, active constraints, and stage transitions may itself require mathematical reasoning or numerical calculations?
- How much solution-relevant information do the matched scene contracts reveal? Can they be constructed without reference solutions?
- How do the authors ensure that relevant and irrelevant field edits are comparable in semantic severity and physical consistency? Could the observed accuracy gaps primarily reflect contradictory instructions rather than selective sensitivity to physical scene information?
3 comments
Reply for R3-Q3: Comparability and physical consistency of X-irr and X-rel edits
We thank the reviewer for raising this important concern. We agree that pairing edits within the same field does not, by itself, guarantee comparable semantic severity, physical plausibility, or consistency with the original question.
To examine these potential confounds, we conducted an item-level preliminary audit of all 284 X-irr/X-rel edit pairs using the existing intervention records. No model reruns were involved in this analysis.
Semantic severity. The two conditions exhibit closely matched aggregate severity. The mean scores were 2.363 for X-irr and 2.444 for X-rel, yielding a mean paired difference of 0.081 (defined as X-rel minus X-irr). The median paired difference was 0, with a 95% bootstrap confidence interval of [-0.025, 0.187] and a Wilcoxon signed-rank p-value of 0.1256. At the item level, 164 of 284 pairs (57.7%) had identical severity scores, while 249 pairs (87.7%) differed by no more than one point. Thus, the preliminary assessment does not indicate a statistically detectable systematic difference in semantic severity between the two edit groups.
Physical plausibility. The results likewise provide no evidence that X-rel edits are systematically less physically plausible. In total, 207/284 X-irr edits (72.89%) and 211/284 X-rel edits (74.30%) were initially assessed as physically plausible. The paired contingency table contained 202 pairs in which both edits were plausible, 5 in which only X-irr was plausible, 9 in which only X-rel was plausible, and 68 in which both required additional assumptions. No edit was classified as outright physically implausible. An exact McNemar test found no significant directional difference (p = 0.4240).
Consistency with the original question. Using the broader binary classification of fully consistent versus not fully consistent (including underdetermined cases), 147 pairs were consistent under both conditions, 18 were consistent only under X-irr, 26 only under X-rel, and 93 under neither. The paired comparison showed no significant directional imbalance (exact McNemar p = 0.2912). Under the stricter direct-contradiction criterion, 9 X-irr edits (3.17%) and 11 X-rel edits (3.87%) were flagged. We note that, in the current automated prescreen, this strict contradiction flag is mechanically derived from the highest question-inconsistency score; we therefore treat these as one indicator rather than independent measures.
Illustrative example. Consider the beam-oscillation problem in PHYBench item 100. The original question explicitly instructs the solver to neglect the beam's own mass and friction. The paired X-irr edit adds that the beam is electrically neutral, a physically possible property that has nothing todo with the mechanical force balance or oscillation derivation. The X-rel edit instead changes the assumption to retain the beam's own mass and friction. This also describes a physically possible system, but directly contradicts the stated problem and changes its inertia and dissipation assumptions. This example illustrates why physical plausibility and question consistency should be assessed separately: both edits can describe physically possible systems, while only one contradicts the original problem setup.
Taken together, these preliminary results do not support the worry that the X-rel condition systematically introduces more severe, less physically plausible, or more contradictory edits. They therefore strengthen the interpretation of the original 23.09-percentage-point accuracy difference as an observed contrast between interventions that do and do not modify commitments used by the registered solution.
Reply for R3-Q2: Solution-relevant information in Gold contracts and their construction
We thank the reviewer for raising the important distinction between making physical scene commitments explicit and revealing solution-specific information.
We have conducted an item-level Gold-information audit on the 120-question paired-inference subset. All 120 contracts passed the overall audit. The target-quantity and state-and-interactions fields were directly supported by the problem statements in all audited cases. The remaining fields—system boundary, constraints and independent components, stages and transitions, and assumptions and approximations—often required partial inference from standard physical knowledge. Importantly, no field was classified as containing solution-specific information unsupported by the question and standard domain knowledge.
We therefore distinguish three sources of information in a scene contract: (1) facts explicitly stated in the problem, (2) reasonable physical inferences based on standard domain knowledge, and (3) solution-specific information that would require access to a particular derivation or answer. Our audit found the first two categories, but no instances of the third in the 120 audited contracts.
Regarding construction provenance, the contracts were drafted in a separate structured authoring stage and reviewed against the problem statements and registered solutions. We therefore do not claim that the original authoring process was fully blind to reference solutions. Instead, our audit evaluates the content of the frozen contracts: whether their physical commitments are independently supportable from the problem statement together with standard physical knowledge, rather than relying on solution-specific information.
The contract is intended to specify the physical scene—not to provide the governing equation, intermediate derivation, numerical substitution, or final answer. We have clarified this distinction and the scope of the 120-item audit in the revised manuscript.
Reply for R3-Q1
R3-Q1: Why is the contract considered pre-derivation?
We surely agree that the term "pre-derivation" could be misunderstood as implying an absence of reasoning. This is not our intended meaning. Our operational definition is pre-equation rather than pre-reasoning.
Constructing a physical scene can itself require qualitative physical reasoning, domain knowledge, and judgments about system boundaries, interactions, constraints, and assumptions. The distinction is that the contract records these physical modeling commitments before formal mathematical equations are established. It does not aim to eliminate reasoning from scene construction, but to distinguish scene-level modeling from subsequent equation formulation, mathematical derivation, and numerical calculation.
We will clarify this definition and its operational boundary in the revised manuscript.
We thank the reviewers for all the points raised by them. We will provide responses to the remaining comments that required experiment results as evidence at a later stage.