acceptodds
Under review as a conference paper at ICLR 2027

Before the First Equation: Selective Scene Control in Large Language Model Physics Reasoning

Abstract

We study how a solver's description of a physics scene shapes its answer. Across 284 questions from PHYBench, SciBench, and UGPhysics, four fixed model interfaces, and three repeated calls per condition, a question-matched six-field contract reaches 94.22% judged accuracy. This is +8.27 points above a no-contract scene baseline collected in a separate run (85.94%; descriptive). Within a shared response run, the matched contract exceeds a mismatched contract from another question by 74.15 points (Holm-adjusted paired sign-flip ). Its benefit depends on correspondence: a one-field solution-relevant edit in the contract yields 56.19%, compared with 79.28% for an edit designated irrelevant to the registered solution in the contract, a 23.09-point paired gap (Holm-adjusted ). The sensitivity is selective but not exclusive; even the designated irrelevant edit has a cost. In a same-call, solving with designed general six-direction self-check reaches 93.57%, versus 85.71% for direct solving (+7.86 points across separate runs, descriptive). Judgments use condition-specific mixtures of programmatic checks, model-assisted judging, and human QC. A well-matched physical description, and potentially an inline check of one's own solve, can support more accurate physics answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, physics reasoning, scientific reasoning, reasoning evaluation, causal interventions, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KkKCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the influence of structured physical scene descriptions on LLM physics reasoning. The authors introduce a six-field Scene Contract and evaluate matched, mismatched, and single-field perturbed contracts across three physics benchmarks and four models. The paper also examines whether inline self-checking improves final-answer accuracy.

Critical strengths
  1. The paper highlights physical scene modeling as an important component of physics problem-solving.
  2. The matched, mismatched, and field-level perturbation experiments provide a structured approach to evaluating sensitivity to physical descriptions.
  3. The evaluation covers multiple benchmarks and models, with statistical analyses supporting the observed performance differences.
Critical weaknesses
  1. Identifying physical states, constraints, and stage transitions often requires substantial reasoning and computation, constituting a core challenge in complex physics problems. Providing a complete, correct six-field scene contract may bypass these essential reasoning steps, raising concerns about the validity of this experimental setup.
  2. The matched scene contracts are constructed using reference solutions, potentially providing privileged modeling information that simplifies the original tasks. The reported gains may therefore overstate the contribution of scene information alone.
  3. Replacing an entire contract with one from another problem may introduce severe contradictions, while relevant and irrelevant field edits may differ in semantic severity. The observed performance drops therefore do not necessarily demonstrate a distinctive physical scene reasoning mechanism.
  4. Only final-answer accuracy is evaluated, without examining intermediate physical modeling decisions. Additionally, the inline self-checking comparison lacks matched-run and compute-controlled baselines.
Questions for authors
  1. How do the authors justify treating the six-field contract as a pre-derivation scene representation when identifying physical states, active constraints, and stage transitions may itself require mathematical reasoning or numerical calculations?
  2. How much solution-relevant information do the matched scene contracts reveal? Can they be constructed without reference solutions?
  3. How do the authors ensure that relevant and irrelevant field edits are comparable in semantic severity and physical consistency? Could the observed accuracy gaps primarily reflect contradictory instructions rather than selective sensitivity to physical scene information?
3 comments
AuthorsOpenReview ↗

Reply for R3-Q1

R3-Q1: Why is the contract considered pre-derivation?

We surely agree that the term "pre-derivation" could be misunderstood as implying an absence of reasoning. This is not our intended meaning. Our operational definition is pre-equation rather than pre-reasoning.

Constructing a physical scene can itself require qualitative physical reasoning, domain knowledge, and judgments about system boundaries, interactions, constraints, and assumptions. The distinction is that the contract records these physical modeling commitments before formal mathematical equations are established. It does not aim to eliminate reasoning from scene construction, but to distinguish scene-level modeling from subsequent equation formulation, mathematical derivation, and numerical calculation.

We will clarify this definition and its operational boundary in the revised manuscript.

We thank the reviewers for all the points raised by them. We will provide responses to the remaining comments that required experiment results as evidence at a later stage.

AuthorsOpenReview ↗

Reply for R3-Q3: Comparability and physical consistency of X-irr and X-rel edits

We thank the reviewer for raising this important concern. We agree that pairing edits within the same field does not, by itself, guarantee comparable semantic severity, physical plausibility, or consistency with the original question.

To examine these potential confounds, we conducted an item-level preliminary audit of all 284 X-irr/X-rel edit pairs using the existing intervention records. No model reruns were involved in this analysis.

Semantic severity. The two conditions exhibit closely matched aggregate severity. The mean scores were 2.363 for X-irr and 2.444 for X-rel, yielding a mean paired difference of 0.081 (defined as X-rel minus X-irr). The median paired difference was 0, with a 95% bootstrap confidence interval of [-0.025, 0.187] and a Wilcoxon signed-rank p-value of 0.1256. At the item level, 164 of 284 pairs (57.7%) had identical severity scores, while 249 pairs (87.7%) differed by no more than one point. Thus, the preliminary assessment does not indicate a statistically detectable systematic difference in semantic severity between the two edit groups.

Physical plausibility. The results likewise provide no evidence that X-rel edits are systematically less physically plausible. In total, 207/284 X-irr edits (72.89%) and 211/284 X-rel edits (74.30%) were initially assessed as physically plausible. The paired contingency table contained 202 pairs in which both edits were plausible, 5 in which only X-irr was plausible, 9 in which only X-rel was plausible, and 68 in which both required additional assumptions. No edit was classified as outright physically implausible. An exact McNemar test found no significant directional difference (p = 0.4240).

Consistency with the original question. Using the broader binary classification of fully consistent versus not fully consistent (including underdetermined cases), 147 pairs were consistent under both conditions, 18 were consistent only under X-irr, 26 only under X-rel, and 93 under neither. The paired comparison showed no significant directional imbalance (exact McNemar p = 0.2912). Under the stricter direct-contradiction criterion, 9 X-irr edits (3.17%) and 11 X-rel edits (3.87%) were flagged. We note that, in the current automated prescreen, this strict contradiction flag is mechanically derived from the highest question-inconsistency score; we therefore treat these as one indicator rather than independent measures.

Illustrative example. Consider the beam-oscillation problem in PHYBench item 100. The original question explicitly instructs the solver to neglect the beam's own mass and friction. The paired X-irr edit adds that the beam is electrically neutral, a physically possible property that has nothing todo with the mechanical force balance or oscillation derivation. The X-rel edit instead changes the assumption to retain the beam's own mass and friction. This also describes a physically possible system, but directly contradicts the stated problem and changes its inertia and dissipation assumptions. This example illustrates why physical plausibility and question consistency should be assessed separately: both edits can describe physically possible systems, while only one contradicts the original problem setup.

Taken together, these preliminary results do not support the worry that the X-rel condition systematically introduces more severe, less physically plausible, or more contradictory edits. They therefore strengthen the interpretation of the original 23.09-percentage-point accuracy difference as an observed contrast between interventions that do and do not modify commitments used by the registered solution.

AuthorsOpenReview ↗

Reply for R3-Q2: Solution-relevant information in Gold contracts and their construction

We thank the reviewer for raising the important distinction between making physical scene commitments explicit and revealing solution-specific information.

We have conducted an item-level Gold-information audit on the 120-question paired-inference subset. All 120 contracts passed the overall audit. The target-quantity and state-and-interactions fields were directly supported by the problem statements in all audited cases. The remaining fields—system boundary, constraints and independent components, stages and transitions, and assumptions and approximations—often required partial inference from standard physical knowledge. Importantly, no field was classified as containing solution-specific information unsupported by the question and standard domain knowledge.

We therefore distinguish three sources of information in a scene contract: (1) facts explicitly stated in the problem, (2) reasonable physical inferences based on standard domain knowledge, and (3) solution-specific information that would require access to a particular derivation or answer. Our audit found the first two categories, but no instances of the third in the 120 audited contracts.

Regarding construction provenance, the contracts were drafted in a separate structured authoring stage and reviewed against the problem statements and registered solutions. We therefore do not claim that the original authoring process was fully blind to reference solutions. Instead, our audit evaluates the content of the frozen contracts: whether their physical commitments are independently supportable from the problem statement together with standard physical knowledge, rather than relying on solution-specific information.

The contract is intended to specify the physical scene—not to provide the governing equation, intermediate derivation, numerical substitution, or final answer. We have clarified this distinction and the scope of the 120-item audit in the revised manuscript.

Reviewer_9WTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical study of how the physical scene description given to an LLM affects its answers to physics problems. The authors introduce an equation-free “scene contract” with six fields: target quantity, system boundary, state and interactions, constraints and independent components, stages and transitions, and assumptions and approximations. Models are instructed to treat the contract as binding. The study uses 284 questions from PHYBench, SciBench, and UGPhysics, four model interfaces, and three calls per condition. Answers are graded by programmatic checks, a blinded LLM judge, and human QC. With the question-matched contract, accuracy reaches 94.22%, compared with 85.94% without a contract in a separate run. Replacing it with a contract from another question drops accuracy to 20.07%. Editing a single solution-relevant field gives 56.19%, while editing a field designated as irrelevant to the solution gives 79.28%. The authors conclude that the benefit depends on scene–question correspondence and that models are selectively, but not fully, robust to local edits. As a secondary result, an in-call six-aspect self-check reaches 93.57%, compared with 85.71% for direct solving, although this comparison is across separate runs and is reported as descriptive only.

Critical strengths

The experimental design is carefully controlled. The four main scene conditions share one response run, as well as the same contract format and solve instruction. Substituting a full contract and editing a single field therefore probe two different things: whether the scene belongs to the question, and how stable the answer is under a local change. Separating relevant from irrelevant edits adds a further distinction that a single “perturbed prompt” score would miss.

The statistical reporting is careful and honest. Comparisons are paired at the item level, the two confirmatory endpoints are fixed in advance with Holm correction, and the paper clearly marks the cross-run comparisons as descriptive. The per-model breakdown (Table 5) also shows that interfaces with similar total losses can split them very differently between the two edit types, which is a useful observation for evaluation design.

Critical weaknesses

The main concern is whether the matched contracts leak solution information. The contracts were drafted with LLM assistance and reviewed against each question’s registered solution. Appendix B states that the audit of whether a Gold contract introduces solution-relevant information absent from the question “remains incomplete.” Without this audit, the +8.27-point gain of G over D_scene cannot be attributed to scene specification rather than to hints about the solution. In addition, the contracts are always supplied by the experimenters, and the paper does not test whether models can produce a correct contract themselves, so the practical takeaway is unclear.

The design of the contract interventions also limits how the results can be interpreted. The prompt instructs the model to treat the contract as binding, so the 74.15-point drop under S-primary mainly shows that models follow this instruction, not that they are sensitive to scene correspondence. A non-binding condition would make this contrast more informative. For the single-field edits, X-irr modifies only one field per item, even though a contract often contains several fields outside the registered solution path, and the paper does not explain how that field was chosen. Results are not broken down by field type, although the X-irr–X-rel gap already ranges from 16.33 to 31.92 points across benchmarks. Since which fields count as “irrelevant” depends on the problem type and on a single registered solution, the 14.94-point cost of an irrelevant edit may partly reflect which field happened to be edited. Editing each eligible field separately, or reporting results by field type, would make the selectivity claim more convincing.

The two comparisons most relevant in practice, G versus D_scene and AU-inline versus D_clean, are both cross-run and reported only as descriptive. Table 7 also shows that judgment sources differ substantially across conditions: AU-inline relies mostly on model-judge outcomes, while D_clean relies mostly on human-QC labels. The direction of the resulting bias is unknown, and an interleaved rerun with a common adjudication procedure would be needed before these gains can be interpreted.

Questions for authors

Can the audit of solution leakage in the Gold contracts be completed? If so, how does G compare with D_scene when it is restricted to contracts confirmed to contain no solution-relevant information beyond the question? How was the edited field chosen for X-irr when several fields lie outside the registered solution path? Could you report X-irr and X-rel results by field type? The paper motivates the study by asking whether errors begin in scene construction or in the subsequent derivation, but all contracts are supplied by the experimenters. Could you have the model first write its own six-field contract before solving, and compare it with the Gold contract? Reporting which fields are most often wrong, and how field-level errors relate to final-answer errors, would directly address the question raised in the introduction.

3 comments
AuthorsOpenReview ↗

Reply for R2-Q2: Field selection and field-wise X-irr/X-rel results

We thank the reviewer for requesting greater transparency about the edit construction procedure.

After auditing the validator and intervention manifest, we clarify that X-irr and X-rel were paired at the item level. For each question, both conditions modified the same one of the six contract fields and started from exactly the same Gold field value. Each edit replaced the complete field value, while the other five fields remained byte-identical.

Field selection was performed through item-specific authoring based on the registered solution path, rather than through a global random field-selection procedure. We will explicitly describe this construction process in the revised manuscript.

We also provide the field-wise results across all 284 questions:

FieldQuestionsX-irr (%)X-rel (%)
Assumptions and approximations10578.9758.97
State and interactions8782.7656.03
Constraints and independent components4174.1950.41
System boundary3077.5062.22
Stages and transitions1578.3353.89
Target quantity680.5625.00

The X-irr advantage is observed across all six field categories.

We will add this field-wise breakdown to the supplementary material. We will also clarify that the field assignments were manually authored on an item-specific basis, rather than randomly sampled.

Here's a representative paired edit example: Illustrative example. Consider the beam-oscillation problem in PHYBench item 100. The original question explicitly instructs the solver to neglect the beam's own mass and friction. The paired X-irr edit adds that the beam is electrically neutral, a physically possible property that has nothing todo with the mechanical force balance or oscillation derivation. The X-rel edit instead changes the assumption to retain the beam's own mass and friction. This also describes a physically possible system, but directly contradicts the stated problem and changes its inertia and dissipation assumptions. This example illustrates why physical plausibility and question consistency should be assessed separately: both edits can describe physically possible systems, while only one contradicts the original problem setup.

AuthorsOpenReview ↗

Reply for R2-Q1: Gold-contract information audit and matched-scene performance

We thank the reviewer for emphasizing the importance of auditing potential solution leakage in the Gold contracts.

We have now completed an item-level Gold-information audit for the 120-question paired-inference subset. All 120 audited contracts passed the overall audit. The target-quantity and state-and-interactions fields were directly supported by the question in all audited cases. The remaining four fields—system boundary, constraints and independent components, stages and transitions, and assumptions and approximations—often required partial inference from standard domain knowledge. Importantly, no field was classified as containing solution-specific information beyond what was supported by the question together with standard physical knowledge.

We distinguish question-supported information from reasonable domain-based inference: the contract is not intended to reproduce only the literal wording of the question, but to make explicit the physical commitments needed to interpret it. Such inference is part of scene construction, rather than the provision of a worked solution.

To directly address the requested comparison, we additionally compare G and D-scene on the 120 audited questions. Both conditions were collected in the same response run using the same scene interface, with the presence of the matched Gold contract as the key difference. The item-paired effect is 5.61% percentage points. You may read more detail in our revised paper.

Thus, the observed matched-scene advantage persists when restricting the analysis to contracts that passed the Gold-information audit. We have clarified the audit scope and added the corresponding results to the revised manuscript and supplementary materials.

AuthorsOpenReview ↗

Reply for R2-Q3: Autonomous scene construction before solving

We thank the reviewer for this constructive suggestion. We have added a self-generated scene condition to the Clean-150 evaluation to examine whether models can construct a physical scene before solving, rather than relying exclusively on experimenter-authored contracts.

The condition uses a two-call frozen protocol. In the first call, the model receives only the problem and generates its own six-field scene contract. This contract is then frozen. In the second call, the model receives the original question together with its own frozen contract and produces the final solution. Thus, the model cannot revise the first-stage scene representation after seeing the subsequent solving task.

Across the three solver interfaces, the descriptive final-answer accuracies are 85.63% for Self, 83.56% for D-clean, and 89.93% for the expert-authored matched-contract condition G. Self therefore provides a modest improvement over direct solving (+2.07 percentage points), while remaining below G (−4.30 percentage points). The pattern also varies across models: Kimi K3 and GLM-5.3 improve over D-clean, whereas Qwen3.8 is slightly lower.

These results provide an initial comparison between autonomous and externally specified scene construction. They suggest that models can produce useful intermediate scene representations, but that their self-generated representations do not yet consistently match the performance obtained with expert-authored contracts.

We agree that field-level evaluation is necessary to identify which scene components are most frequently misconstructed and how such errors relate to final-answer correctness. The current results report the final-answer outcome of the frozen self-generated-contract pipeline; they do not yet establish a field-level error profile. We will make this distinction explicit and identify it as a limitation and future direction.

Reviewer_7BebOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how a supplied physical-scene description shapes an LLM's physics answer.

Setup. Four interfaces (DeepSeek V4 Flash, GPT-5.6 Sol and Terra, Kimi K3) answer 284 PHYBench, SciBench and UGPhysics items three times per condition. A six-field, equation-free "scene contract", authored outside the solver and declared binding, is appended to each question.

Within-run comparisons. The matched contract (G, 94.22%) is compared, within one run, with:

  • another item's contract (S-primary, 20.07%);
  • a one-field edit designated solution-relevant (X-rel, 56.19%);
  • a one-field edit designated irrelevant (X-irr, 79.28%).

Separately collected conditions give two descriptive comparisons:

  • a no-contract scene baseline (85.94%, 8.27 points below G);
  • a same-call self-check prompt (93.57%) against direct solving (85.71%).

Judgments mix programmatic checks, a model judge, human-reviewed fallback labels and human QC.

Critical strengths
  1. The within-run design is clean and candidly reported.

    • The four scene conditions share one run and are paired by item, model and call.
    • Inference is item-level, with Holm correction over two endpoints (p.4 L205-215).
    • Neither confirmatory endpoint uses unreviewed model-judge labels (Table 7). Both signs hold without their 22-24% human-reviewed "codex_fallback" labels (supplement data/judgment_source_counts.csv).
    • Every count and rate in Tables 2 to 7 recomputes exactly from the released ledger (data/outcomes.csv).
    • The leakage-audit gap (L527-528) and the anomalous earlier wave (L559-561) are disclosed.

    This matters because the measurements are trustworthy; the problems lie in what they establish.

  2. The matched / mismatched / one-field sequence is a reusable evaluation probe (L374-377), released with its contracts and edits. Its least expected result is robust within the run: an edit designated outside the solution path still costs 14.94 points [12.97, 16.96], from 7.2 points (Kimi) to 29.6 points (Terra) by interface (Table 5). This matters because a structured specification can destabilize answers even where its designers certified the registered solution unaffected. Some certified edits contradict the question, e.g. mass 2M for M (PHYBench 336).

Critical weaknesses
  1. The confirmatory results measure compliance with solution-informed oracle contracts, not scene reasoning.

    • The solver must treat the contract as binding and must not reinterpret it (p.10 L506-508). Answers are scored against the original question's reference answer (L193-196).
    • S-primary is 65.87 points below the no-contract baseline (Table 2, cross-run). About 89% of the 74.15-point gap is therefore the cost of a conflicting binding contract, not a benefit of the matched one.
    • Compliance also predicts X-irr > X-rel whenever the designations are right (App. B L522-524; each released X-rel edit states the answer change). Both confirmatory signs therefore follow from obedience.
    • The contracts were drafted with LLM help and reviewed against the registered solution (p.3 L158-160; p.8 L421-423), and the leakage audit is incomplete (L527-528). Read against the public questions, many released contracts leak. In 15 of the 18 items where G beats Dscene by at least 6 of 12 answers, the contract supplies solution content the question lacks (a value, method, intermediate result or trap resolution). These items carry 134 of the 282-row gain; e.g., a 3 mm pupil diameter fixes the answer in UGPhysics GeometricalOptics 5173. PHYBench 256 adopts "the semiclassical bound-state treatment used in the reference solution".
    • No reported condition has the solver write its own contract or receive one as advice, and the paper shows no example contract.
    • One edited field already makes a contract worse than none. X-rel is below Dscene for all four interfaces; X-irr is below it overall and for DeepSeek (by one row), Sol and Terra (Table 4, cross-run).

    This is consequential because the abstract and conclusion (L26-28, L404-406) recommend scene specification for reliability, whereas 94.22% is a ceiling under solution-informed annotation.

  2. The practically meaningful results (+8.27 contract, +7.86 self-check) are cross-run and confounded.

    • An earlier direct wave was anomalously high, for unresolved reasons and by an undisclosed amount (L559-561). Dclean and AU-inline have no collection dates (L472-474).
    • The self-check comparison is adjudicated asymmetrically. AU-inline keeps its 2,517 model-judge positives (79% of its correct rows), while every non-programmatic Dclean row has a QC label. All 396 judge negatives went to human QC, and 177 of them (44.7%) were overturned to correct (+5.19 points; Fig. 2, Tables 2 and 7).
    • No AU-inline label was lowered (Table 7). The judge-positive audit (L198-199) has no reported size or outcome, and the released protocol and code finalize judge positives without one (supplement programmatic_judge/correctness_triage.md L94; physics_scene_eval/correctness_pregrade.py L1051-1069).
    • Under the paper's human-precedence rule (L534-535), the bias therefore favors AU-inline, and "direction ... unknown" (L322) is incomplete. The net bias stays unmeasured, because Dclean's labels have three provenances (L535-538).
    • The paper does not name the judge. The supplement's code pins gpt-5.5 (programmatic_judge/physics_scene_eval/correctness_judge.py L34), but the AU-inline judge file is luna_judge_completed.jsonl (data/source_audit.json); no reliability or self-preference check is reported.
    • Without DeepSeek, whose Dclean labels come from a separate packet, the gain is 5.99 points (Table 4).

    This is consequential because the self-check result is promoted in the abstract and conclusion.

Questions for authors

Clarification (answers could change my assessment):

  1. Please show several contracts in the paper with their questions and solutions (the supplement omits the questions). How often does a contract state something not inferable from the question? Under S-primary and X-rel, how often does an answer follow the contract, answer the original question, or flag the conflict?
  2. Did gpt-5.5 or a "luna" model (GPT-5.6 Luna is a sibling of Sol and Terra) produce the 2,517 AU-inline judge positives? What were the size and outcome of any judge-positive audit, and can its findings replace labels? Are the "luna_draft", "codex_fallback" and "sol_rejudge" sources (data/source_audit.json) model-drafted? Were reviewers blind to model and condition (AU-inline packets are per model; DeepSeek's Dclean packet is "with_condition")?
  3. When, and under which model identifiers, were Dclean, AU-inline and the anomalous wave collected, and what was that wave's accuracy per interface? DeepSeek's Dclean positives come from a "rerun-audit-20260924" ledger: were any DeepSeek Dclean responses collected after 2026-09-10? (The deepseek-v4-flash name changed checkpoint on 2026-07-31 and was rerouted to V4.1 Flash on 2026-09-10; api-docs.deepseek.com/updates.)
  4. Dscene uses the binding scene prompt (configs/experiment_protocol.json): what filled its contract slot? How were the 284 items selected? What are the results on the reserved 100-item clean selection (L568-569) and on S-sens1 and S-sens2? The scene runner expects 4,008 tasks per model, not 3,408 (physics_scene_eval/simple_official_resume.py L27).

Suggestions:

  • Add non-binding and solver-written contract arms in a run shared with G (subsets suffice); the code already has a solver-written condition (C, P-C; physics_scene_eval/runner.py L424).
  • Interleave self-check and direct solving under matched adjudication. Move the three-call majority votes into the paper: Dclean 86.27% and AU-inline 93.93% (data/majority_vote_accuracy.csv).
  • Position the work against Physics Reasoner (COLING 2025) and PhysicsEval (Findings of IJCNLP-AACL 2025), engage PHYBench's Sec. 5.1 error analysis, and cite PHYBench as NeurIPS 2025 Datasets and Benchmarks rather than arXiv (L450-451).
  • Name the contracts and edits in the file lists (L411-416, L584-589), and supply App. D's build_v23.py and ANCHOR_INDEX_20260925.md.

Rating rationale.

Not a clear rejection. The paired design and the bookkeeping are sound, the numbers reproduce from the released ledger, and the limitations are disclosed. Not an acceptance. Both confirmatory signs are what obedience to a binding, solution-informed contract predicts, and many released contracts leak solution content. The practical gains are cross-run, and the self-check gain also carries a one-directional, unaudited adjudication difference. What would move me to weak acceptance:

  • compliance coding of the S-primary and X-rel answers;
  • a non-binding or solver-written arm that gains over no contract within the run;
  • a leakage audit, with G − Dscene on leak-free items;
  • the judge-positive audit outcome, or an interleaved self-check run.
3 comments
AuthorsOpenReview ↗

Reply for R1-Q1&Q4

[1]R1-Q1: Clarification of the six-field scene contract

We thank the reviewer for pointing out that the role of the six-field contract was not sufficiently clarified. Our intention is not to provide additional solution information, but to externalize the physical commitments that an expert solver would establish before formal equation derivation.

To illustrate, consider a simple Gauss's law problem (scibench Question 23.7) asking for the net electric flux through a cube enclosing a point charge. The six-field scene contract here specify (1) the requested quantity (net flux through the entire closed surface), (2) the system boundary (the cube's closed surface), (3) the physical state and interactions (the enclosed point charge and its electric field), (4) the relevant constraints (the enclosed charge and closed Gaussian surface), (5) the coarse-grained modeling sequence (identifying the enclosed charge before evaluating the flux), (6) the idealizations (the electrostatic setting and the assumed medium).

Importantly, the contract does not provide the mathematical expression of Gauss's law, numerical substitutions, intermediate calculations, or the final answer. Instead, it makes explicit the physical interpretation and modeling commitments that the solver must use. We acknowledge that this constitutes task-specific scene-level guidance; the distinction we intend to draw is between such guidance and providing a worked solution.

We will include this example and clarify the distinction between question-supported information, standard domain-based physical inference, and potentially unsupported solution-specific information. We will also report the results of a systematic audit of the 284 contracts to quantify the extent of any unsupported or solution-specific content.

[2] R1-Q4: D_scene prompt

We will include the complete D_scene system and user prompts in the supplementary material, together with the output-format specification. D_scene uses the same scene-interface solving instruction as G, with the key difference that no six-field contract is appended. This will make the baseline interface directly verifiable.

We thank the reviewers for all the points raised by them. We will provide responses to the remaining comments that required experiment results as evidence at a later stage.

AuthorsOpenReview ↗

Reply for R1-Q2: Provenance and reliability of AU-inline judgments

We thank the reviewer for raising this concern. After checking the original execution scripts and logs, we confirm that the 2,517 judge-positive decisions were produced by GPT-5.6 Luna Max (with agent), rather than GPT-5.5.

Of the 3,408 AU-inline responses, 2,913 were submitted to the model judge: 2,517 received positive judgments and 396 received negative judgments. All 396 judge-negative cases were subsequently routed to human QC, comprising 99 DeepSeek, 103 GPT-5.6 Sol, 97 GPT-5.6 Terra, and 97 Kimi responses.

We also conducted a human audit of 250 sampled judge-positive cases. Human reviewers upheld 242 decisions and overturned 8, corresponding to a 96.8% agreement rate within this audit sample. Importantly, the eight overturned decisions were incorporated into the final correctness computation.

We thank the reviewer for raising the concern that the judge model came from the same model-family of some tested model. We acknowledge a potential model-family dependence when GPT-5.6 Luna Max evaluates responses from GPT-5.6 Sol and Terra. Although the judge was blinded to solver identity and experimental condition, blinding alone does not establish model-family independence. We therefore report the human-audit results explicitly rather than claiming fully independent model-based verification.

As additional evidence, we conducted a supplementary evaluation using GLM-5.3 and Kimi K3 on a curated set of 150 questions with verified problem statements and reference answers. We retained the same staged adjudication procedure, using programmatic judge firstly, and then using GPT-6 Luna Max as the judge, followed by human QC of unresolved or rejected cases and an audit of sampled positive judgments. The observed effects remain consistent with our main findings. We regard this as additional cross-solver validation, rather than a fully independent judge-family validation.

Finally, we'd like to clarify the auxiliary provenance labels in the supplementary materials: luna_draft denotes Luna-generated review judgments following programmatic judge rejection; codex_fallback denotes the agent-assisted fallback pathway using Codex within the Luna Max workflow; and sol_rejudge denotes rejudgment of responses produced by GPT-5.6 Sol.

We will provide the corresponding execution details where available.

AuthorsOpenReview ↗

Reply for R1-Q3: Collection provenance and the anomalous earlier wave

We thank the reviewer for requesting clarification of the collection history.

We distinguish the historical anomalous wave associated with the earlier cross-call AU procedure from the experiments reported in the revised manuscript. The legacy AU procedure used frozen answers from an earlier direct-solving wave as its parent. However, the per-interface accuracies of that wave cannot be reliably reconstructed from the available records, and the cause of the anomaly remains unresolved. We therefore do not report inferred per-interface accuracies or use this wave for any parent-to-review accuracy comparison.

Importantly, this anomalous wave is not part of the current Clean-120 experiments and does not contribute to any of the revised paper's primary results. The current experiments were conducted as a separate evaluation on the curated dataset.

The solver interfaces used in the current experiments are Kimi K3 (kimi-k3), GLM-5.3 (glm-5.3-spot), and Qwen3.8 (qwen3.8-2.4t-a95b-spot).

We have revised the provenance documentation to distinguish the original exploratory runs from the Clean-120 evaluation, and to explicitly mark unrecoverable historical metadata rather than infer it from transferred file timestamps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.