acceptodds
Under review as a conference paper at ICLR 2027

GroundForm: Learning to Ground Autoformalization with Verified Hindsight Credit and Growing Memory

Abstract

Autoformalization requires a model to reconcile natural-language intent, visual evidence, and domain knowledge in a single precise statement. We present Ground- Form, an agent that formalizes both textual mathematics and multimodal physics and learns which knowledge to retrieve along the way. It is trained with Verified Hindsight Credit (VHC): the formalizations that pass verification within a rollout group serve as targets, and a frozen reference model scores how much closer each tool interaction brings the agent to them, so that intermediate retrieval decisions receive their own credit. In addition, GroundForm maintains two verified memo- ries that grow during training. Mathematical experience can be retrieved in both domains, and physical experience supplies laws and conventions missing from the static libraries. On eight benchmarks, GroundForm reaches 80.8% average semantic accuracy in mathematics and 58.2% in physics, 57.6 and 43.6 points above its untrained Qwen3.5-9B backbone. Interventions on tools, memory, and images help locate the source of these gains. VHC increases training time per step by 16.8%, yet it needs 41% fewer GPU-hours to reach the final PhyX-AF accuracy of outcome-only training.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multimodal Reasoning, Reinforcement learning, Multimodal Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AeXpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

GroundForm is a tool-using agent on Qwen3.5-9B that turns textual math problems and image-plus-text physics problems into Lean 4 statements.

While formalizing, it can search Mathlib, a physics library, two growing memories and the web.

Training is SFT on trajectories distilled from strong closed models, then GRPO with Verified Hindsight Credit (VHC).

In VHC, statements in a rollout group that compile and pass an LLM judge become targets, and a frozen reference model scores how much each tool call raises the likelihood of those targets, so every retrieval step gets its own credit. The two memories store verified math and physics experience during training and are used at test time.

The evaluation covers four math and four physics benchmarks, reporting 80.8% math and 58.2% physics semantic accuracy, a shared-backbone ladder (GRPO, shuffled VHC, VHC, full model), and analyses of tools, memory and images. VHC is a neat way to get turn-level credit without gold formalizations, though it is close to recent information-gain style credit. The physics results have data and judge problems that need fixing before the numbers can be trusted.

Critical strengths
  1. VHC is a simple and well-fitted answer to a real problem: how to credit intermediate retrieval steps when there's no gold formalization. The shared-backbone ladder is well designed, and the shuffled-credit control falls below GRPO while real VHC gains about 4.5 points in math. This matters because it suggests that where the credit lands matters, not just having a denser signal, and other agent RL work could reuse this.

  2. The image analysis gives a clear lesson for multimodal formalization. Success depends heavily on whether what the image shows is actually written into the Lean statement, and masking the image cuts image-necessary accuracy sharply. This is useful to anyone building multimodal autoformalizers, independent of the training recipe.

Critical weaknesses
  1. The physics evaluation looks contaminated. Table 7 puts 'PhyX full test' (3,000 items), 'Physlib entries collected from evaluation problems' and 'PhysicsEval (train + test)' in the RL pool, while 1,000 PhyX problems are used for evaluation, and I could not find any decontamination step. The memories are built from accepted training trajectories and used at test time. With static libraries only, the full model scores about the same as VHC (Table 9), so the whole memory gain comes from test-time retrieval, which is exactly where lookup of test items would show up. This matters because it affects the 58.2% physics number, the memory and transfer claims and the PhyX-AF efficiency result. Fixing it needs retraining and re-evaluation on clean data, not just a clarification.

  2. The physics metric isn't shown to measure faithful formalization, and the comparisons and positioning are incomplete. The physics judge is a closed LLM with a lenient rule, it is validated only on math, and accepted outputs can be near-tautological: the standing wave statement in App. G assumes d = 1/2 * (v/f) as a hypothesis and concludes v = f * 2 * d. Math baselines are copied single-pass ReForm numbers (with ReForm-32B left out), physics has only Gemini-2.5-Pro, and the SFT teachers are never evaluated with the same tools. VHC is also close to IGPO (turn-level information gain, ICLR 2026) and likelihood rewards like RLPR, and retrieval-grounded autoformalizers (RAutoformalizer, DRIFT, Aria) aren't cited. So both the size of the gains and the novelty are less clear than the abstract suggests.

Questions for authors
  1. Are the 1,000 evaluated PhyX problems, and the PhyX-sourced items of PhyX-AF, inside 'PhyX full test' or 'PhyX-100'? What exactly are the 437 'Physlib entries collected from evaluation problems', and can any memory or PhysSearch entry come from an evaluation item? Please report physics results after removing evaluation items from the RL data, memories and search index, with web search blocked from benchmark sources. This is the main thing that could change my score.
2 comments

Q1: Could the physics evaluation be contaminated by the training data, memory, the search index, or WebSearch?

A1: Thanks for your comments. Table 7's “PhyX full test 3,000” label is incorrect. The actual split, used in all reported experiments, keeps evaluation, SFT, and RL problems disjoint:

DatasetTotalEvalSFTRL
PhyX3,0001,0004501,550
SeePhys2,0001,000661339
PhysUniBench3,3041,0001,1051,199

The 256 PhyX-AF problems are a subset of the 1,000 PhyX evaluation problems and were excluded from SFT and RL. The 437 entries are growing-memory entries distilled from accepted training-set trajectories, not evaluation-derived content, RL samples, or PhysSearch index entries. Neither the frozen evaluation memory nor the PhysSearch index contains evaluation problems or derived content, and benchmark sources are blocked in WebSearch during evaluation. The reported physics results were obtained under exactly this setting:

DatasetSyn (%)Sem (%)
PhyX96.678.1
SeePhys90.951.7
PhysUniBench85.748.2
PhyX-AF95.754.7
Average92.258.2

We will correct Table 7 and document the isolation of each component.

Q2: Does physics Sem measure faithful formalization? Are the baseline comparisons and the related-work positioning sufficient?

A2: Thanks for your comments.

Sem and faithfulness. Following ReForm [1], we use model-judged Sem as a proxy for faithful formalization; it is not equivalent to formal verification of faithfulness. For image-based physics, we validated the judge against human labels on 240 samples (40 original + 200 new) with 82.1% accuracy (72.5% / 84.0%). In training and evaluation, the judge receives the original text, image, and Lean statement. We agree with the reviewer on the Appendix G standing-wave example: once the node-spacing relation is stated as a premise, the conclusion is its algebraic rewriting, and the judge accepted this weak formalization. We will report it as a limitation, temper our Sem claims, and study how to detect such cases.

Tool protocol. To measure how much tools alone help an external model, we gave Gemini-2.5-Pro the same tool interface and call budget as GroundForm: its average math Sem rises from 10.1% to 23.6% (miniF2F 36.9, ProofNet 21.9, Putnam 20.5, AIME25 15.0). The training contribution of VHC is isolated by internal runs sharing the backbone, SFT initialization, and training configuration: VHC improves over GRPO by 4.5 (math) and 4.7 (physics) Sem points, while shuffling credit across tool turns lowers performance below GRPO.

More baselines. Physics, direct generation, avg Syn / Sem (%): GPT-5 75.3 / 47.3; Claude-3.7-Sonnet 62.0 / 34.3; Gemini-2.5-Pro 61.1 / 40.1; Gemini-3-Pro 77.5 / 49.8; Qwen3-VL-235B 66.8 / 33.0; Qwen2.5-VL-72B 55.0 / 24.8; GroundForm 92.2 / 58.2. On PhyX-AF, MMFormalizer obtains 41.1 / 31.9 (GPT-5), 50.7 / 44.9 (Gemini-3-Pro), 24.6 / 19.9 (Gemini-2.5-Pro), 10.7 / 8.4 (Qwen3-VL-235B), 2.9 / 0.7 (Qwen2.5-VL-72B), and 87.0 / 15.0 with our Qwen3.5-9B backbone, versus 95.7 / 54.7 for GroundForm.

Qwen3-8B backbone (math avg Syn / Sem, %): SFT 61.8 / 51.0, GRPO 78.3 / 68.3, GroundForm-8B 84.3 / 76.0, ReForm-8B (reported) 85.5 / 64.3. Our SFT stage alone is below ReForm-8B; the gain comes from subsequent training and system components. Our SFT trajectories come from Claude Opus 5 and Qwen3.7-Plus, whereas ReForm uses a Qwen3-235B-A22B-Thinking multi-agent pipeline, so this is not a fully controlled comparison. We did not reproduce ReForm-32B and will qualify cross-model claims accordingly.

Related work and VHC. IGPO rewards turns by the change in the probability of the correct answer, and RLPR uses the reference answer's token probability as the reward. RAutoformalizer, DRIFT (decomposition with premise retrieval), and Aria (dependency-graph-guided retrieval) use retrieval to aid Lean formalization. VHC shares likelihood-based credit with the former and retrieval-based grounding with the latter. Its specific contribution is a turn-level likelihood-potential difference without a unique gold formalization: candidates in a rollout group that pass Lean compilation and the semantic judge serve as targets, a frozen SFT reference model scores their aggregated likelihood before and after each tool turn, and the difference credits that turn. VHC thus needs no human gold statement but relies on the compiler and the judge. We will add this discussion and narrow our novelty claims accordingly.

[1] ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization

Dear Reviewer,

Thank you again for your time and thoughtful comments on our paper. We have posted a detailed response to your concerns. As the discussion period is coming to an end, we would greatly appreciate it if you could take a look and let us know whether our response has addressed your concerns. We are happy to provide any further clarification if needed.

Best regards, Authors

Reviewer_HucdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GroundForm, which translates natural-language mathematics problems and physics problems with diagrams into Lean 4 formal statements. The goal is to produce statements that both compile and faithfully preserve the original problems’ assumptions and conclusions. Methodologically, the authors propose Verified Hindsight Credit (VHC), which uses verified statements from the same rollout group to assign training credit to intermediate tool interactions, alongside mathematical and physical experience memories that grow during training. Comparisons using the same backbone and ablation studies suggest that both VHC and memory contribute to the performance gains.

Critical strengths
  1. An important and well-defined task. The paper distinguishes between a Lean statement that merely compiles and one that faithfully captures the original problem. This distinction addresses a central challenge in autoformalization: syntactic validity alone does not guarantee that the assumptions and intended conclusion have been preserved.

  2. A valuable combination of methods. Verified Hindsight Credit (VHC) assigns training signals to intermediate tool interactions, while the mathematical and physical memories provide reusable information for retrieval. Together, these components offer a coherent approach to improving how the model grounds its formal statements.

  3. Clear and accessible presentation. The paper is well structured and easy to follow. It introduces the task and motivation before explaining the method, then uses experiments and ablations to examine the main components, making the overall argument straightforward to track.

Critical weaknesses
  1. Evaluation-set isolation has not been established. The RL data include 3,000 problems labeled “PhyX full test,” while the evaluation uses 1,000 PhyX problems. The data also include 437 Physlib entries collected from evaluation problems. The authors should provide a problem-level overlap audit covering both the problems and any derived content, then rerun the evaluation on a cleanly separated dataset.

  2. The physics Sem metric lacks targeted human validation. The judge’s reported 87.5% accuracy comes from a mixed annotated set, while the cross-judge comparison of model results is reported primarily for mathematics. These findings do not directly establish the reliability of semantic judgments for physics problems with diagrams. Moreover, although the method requires checking conditions supplied by an image, the published judge prompt does not clearly show how the original image is provided. The authors should disclose the actual judge requests, not just the text prompt template.

  3. The claims about visual grounding are stronger than the reported counts support. Among the 97 image-necessary problems, visual information was written into the Lean statement in 42 GroundForm trajectories and 40 GRPO trajectories. The image ablations show that both methods depend on the image, but they do not yet demonstrate that GroundForm is better at correctly binding visual constraints in its formal statements.

  4. The frontier-model baselines are not sufficiently up to date. The paper compares GroundForm against GPT-5 and Gemini-2.5-Pro but does not explain why it omits newer models available around the time of the manuscript, such as GPT-5.6 or GPT-6 variants. This limits the strength of its comparisons with contemporary general-purpose models.

Questions for authors

Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.

5 comments

Q1: Are training and evaluation isolated? Where do the 437 entries come from and how are they used? Does PhyX-AF overlap with PhyX?

A1: Thanks for your comments. The source labels in Table 7 were incorrect, and we apologize for the confusion. The actual split, used in all reported experiments, keeps evaluation, SFT, and RL disjoint at the problem level:

DatasetTotalEvalSFTRL
PhyX3,0001,0004501,550
SeePhys2,0001,000661339
PhysUniBench3,3041,0001,1051,199
Total8,3043,0002,2163,088
  • PhyX: “PhyX full test 3,000” is incorrect. 1,550 PhyX problems are used for RL and 450 for SFT; none of the 1,000 evaluation problems appears in either set. The reported results therefore already come from this isolated split.
  • 437 entries: these are growing-memory entries distilled from accepted training-set trajectories. They contain no evaluation problems or derived content, and they are neither RL samples nor static-library entries. The corrected RL set has 185,337 examples (3,088 with images).
  • PhyX-AF: all 256 problems belong to our 1,000 PhyX evaluation problems and have zero overlap with SFT/RL. PhyX-AF is thus a subset of the PhyX evaluation set, not an independent test set.

Problem-level overlap audit:

PairOverlap
Evaluation ∩ SFT0
Evaluation ∩ RL0
Evaluation ∩ sources of memory entries0
PhyX-AF ∩ SFT/RL0
PhyX-AF ∩ PhyX evaluation256 (all)

We will correct Table 7 and document the sources of the evaluation set, training set, and memory.

Q2: Has the physics semantic evaluation been specifically validated? Does the judge actually receive the original image?

A2: Thanks for your comments. We added a targeted human-label validation on image-based physics problems. In addition to the 40 such samples in ConsistencyCheck-1K, we annotated 200 new ones. Agreement between the judge (Qwen3.7-Plus non-thinking) and human labels:

SubsetSamplesAccuracy
Original4072.5%
Newly added20084.0%
Total24082.1%

Accuracy on the original 40 samples is lower than on the full ConsistencyCheck-1K (87.5%), which confirms the reviewer's point that aggregate accuracy overstates reliability on image-based physics. We will therefore report the 82.1% physics-specific figure alongside the aggregate one.

In both the training reward and the final evaluation, the judge receives the original problem text, the original image, and the generated Lean statement, not a textual description of the image. Both stages use the same judge and prompt; the model-assisted protocol follows ReForm (ICLR 2026). We will state this multimodal input explicitly in the revision.

Q3: Does the visual-grounding claim go beyond what the counts and ablations can support?

A3: Thanks for your comments. We agree that “using the image,” “writing visual information into Lean,” and “binding visual constraints correctly” should be distinguished. On the 97 image-necessary problems:

MetricGRPOGroundForm
Trajectories writing visual info into Lean40/97 (41.2%)42/97 (43.3%)
Sem within those trajectories67.5% (27/40)81.0% (34/42)
Sem, full image36.145.4
Sem, key regions occluded14.417.5
Sem, image removed5.26.2

Sem measures the semantic consistency of the whole statement, not visual-constraint binding specifically, and the interventions show that both methods depend on the image. We will revise the claim to: both methods write visual information into Lean at similar rates; GroundForm attains higher semantic accuracy on these trajectories; both depend on conditions provided by the image. We will not claim that GroundForm binds visual constraints more accurately.

Q4: Could the authors compare with newer frontier models such as GPT-5.6?

A4: Thanks for your comments. We evaluated GPT-5.6 alongside GPT-5 (Sem, %):

DomainBenchmarkGPT-5GPT-5.6
MathminiF2F66.071.1
MathProofNet44.652.9
MathPutnam45.853.9
MathAIME2513.326.3
MathAverage42.451.1
PhysicsPhyX65.070.3
PhysicsSeePhys49.056.7
PhysicsPhysUniBench40.049.0
PhysicsPhyX-AF35.044.8
PhysicsAverage47.355.2

GPT-5.6 improves over GPT-5 by 8.7 (math) and 7.9 (physics) points. GroundForm reaches 80.8% in math and 58.2% in physics, 29.7 and 3.0 points above GPT-5.6. We will add these results to the revision.

Thank you for your response and the additional experiments. I have one follow-up question. In your response to Q3, you state, “We will not claim that GroundForm binds visual constraints more accurately.” Could you clarify what advantages GroundForm offers over GRPO and why a separate method is needed?

Since GroundForm is not claimed to bind visual constraints more accurately, what advantages does it offer over GRPO, and why is a separate method needed?

Thank you for the follow-up. The narrowed claim concerns only visual perception, which GroundForm does not target. GroundForm targets a different bottleneck in tool-augmented autoformalization: assigning credit to individual retrieval decisions and accumulating verified knowledge. VHC does not replace GRPO. It keeps GRPO's clipped objective and adds a turn-level term to the advantage (Eq. 6).

Why GRPO alone is insufficient. GRPO gives every token of a trajectory the same outcome advantage. With up to nine tool calls per problem, this reinforces redundant searches in successful trajectories and penalizes correct retrievals in failed ones. A standard process reward would require a gold formalization, but autoformalization has none, because several Lean statements can be equally faithful. VHC uses the formalizations in the rollout group that pass Lean compilation and the semantic judge as targets. It credits each tool turn by how much the turn raises a frozen reference model's likelihood of those targets. GroundForm also maintains verified memory that grows during training; GRPO has no such mechanism.

Controlled comparison. All runs below use the same Qwen3.5-9B SFT checkpoint, training data, 160 steps, and rollout configuration.

GRPOVHCGroundForm
Math avg Sem (%)73.778.280.8
Physics avg Sem (%)49.954.658.2
Steps to reach GRPO's final PhyX-AF Sem (44.5%)1608159
GPU-hours to reach it415245178
Physics tool calls per success12.048.006.40
Physics tokens per success21.6k14.3k12.1k
Math MathSearch calls per success5.564.213.66
  1. Accuracy. GroundForm exceeds GRPO on every physics dataset, by 6.8 to 10.2 points. On a Qwen3-8B backbone it exceeds GRPO by 7.7 points (68.3 → 76.0). The ranking is unchanged under an alternative math judge (72.8 / 77.2 / 79.8).
  2. The gain comes from turn alignment. Permuting the same VHC credit values across tool turns (Shuffled VHC) gives 72.9 / 49.2, below GRPO. The extra signal therefore helps only when it is attached to the right turn. Aggregating over the full accepted set also beats a single target (PhyX-AF 50.4 vs. 47.6 random and 48.3 most-likely; Table 3), which supports the design for the setting without a gold formalization.
  3. Efficiency. VHC costs 16.8% more time per step, but it reaches GRPO's final PhyX-AF accuracy with 41% fewer GPU-hours. At inference, GroundForm uses 47% fewer tool calls and 44% fewer generated tokens per physics success.
  4. Retrieval behavior. On PhyX-AF, WebSearch use falls from 96% under GRPO to 24%, MathSearch use rises from 36% to 47%, and the reuse rate of retrieved declarations rises from 38.6% to 60.1%. Removing PhysSearch costs 12.5 Sem points. Together, these results show that the learned policy relies more on local formal declarations.
  5. Memory. With the trained policy held fixed, removing all memory lowers physics Sem from 58.2% to 54.8%.

Relation to the visual results. On the 97 image-necessary problems, the two methods write visual information into Lean at similar rates (41.2% vs. 43.3%). When they do, GroundForm's statements are semantically correct more often (81.0% vs. 67.5%), and its full-image Sem is higher (45.4% vs. 36.1%). Because both methods write visual information into Lean at similar rates, we interpret this gap as coming from the non-visual parts of grounding that VHC and memory target, such as choosing the right law, library declaration, and convention, rather than from diagram reading. In the paper, "grounding" means linking the problem to external knowledge and formal objects (Sec. 1). We will state explicitly that visual perception comes from the shared backbone and is not a claimed advantage. We will also restate the contribution as turn-level credit for retrieval in autoformalization without a gold formalization, plus verified growing memory.

Reviewer_WDcJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents GroundForm, a Lean 4 autoformalization agent for textual math problems and multimodal (diagram-bearing) physics problems. The agent can call three retrieval tools: MathSearch over Mathlib, PhysSearch over Physlib, and WebSearch. Training runs in two stages:

  • SFT: about 10K examples distilled from frontier models.
  • RL with Verified Hindsight Credit (VHC): within each rollout group, statements that pass both the compiler (F) and an LLM semantic judge (C) form a target set Y⁺. A potential Φ is the length-normalized log-sum-exp likelihood of Y⁺ under a frozen SFT reference model, conditioned on the context at each tool boundary. The turn-level reward ΔΦ is added to the GRPO outcome advantage.

The agent also keeps two growing verified memories (math and physics), abstracted from accepted training trajectories. Standard evaluation uses the final frozen memories.

Critical strengths

S1 VHC is a clean and well-motivated idea for credit assignment when no unique gold answer exists : Using the group's own verified statements as hindsight targets, and assigning credit to tool calls via reference-likelihood differences, is a sensible answer to a real problem: a statement can compile and still be unfaithful. The method is fully specified (Eq. 3–6), and App. A correctly notes that log-mean-exp only shifts Φ by a constant and that the return telescopes. This is the part of the paper I would most like to see survive in a revised version.

S2 The Shuffled-VHC control is genuinely informative : It keeps the multiset and the sum of the turn differences Δ but permutes which turn receives each one. Shuffled VHC falls below GRPO (72.9 / 49.2), while VHC rises above it. This is direct evidence that aligning credit to specific turns, not just adding extra reward mass, is what helps.

S3 Well-controlled internal comparisons : All RL variants start from the same SFT checkpoint with the same 160 steps and rollout configuration. The tool and memory ablations hold the GroundForm policy fixed (Table 4), which separates inference-time components from training effects.

Critical weaknesses

W1. The test-time memory can act as an answer key, and it accounts for all of GroundForm's advantage over VHC. Memory entries are abstracted from accepted training trajectories, and evaluation uses the final frozen memory (Sec. 3.3). Because the training data contains evaluation problems, the memory can hold judge-approved formalizations of those same problems. In Table 9, the GroundForm policy without memory (Static-only, 54.8) is essentially equal to VHC (54.6). So the paper's second core contribution gets its entire gain from the component most exposed to leakage. GRPO and VHC were also trained on the contaminated data, but they have no way to retrieve those statements at test time.

W2. External comparisons are not matched. The external baselines' results "follow the evaluation in ReForm" (Table 1), apparently without tools, memory, or GroundForm's 20,480-token / 9-call / top-5 retrieval budget. They also use a different backbone (Qwen3.5-9B vs. ReForm-8B) and a weaker distillation teacher; the same-backbone SFT already reaches 57.6 vs. ReForm-8B's 64.3. For physics, the only external baseline is Gemini-2.5-Pro, under an unspecified protocol, and MMFormalizer, which proposed PhyX-AF, is missing.

Questions for authors

1. Are the 1,000 PhyX evaluation problems included in the "PhyX full test 3,000" RL data? If you deduplicated, please describe the procedure and the remaining overlap. Where were the 437 "Physlib entries collected from evaluation problems" used (static library L_p, memory, or both)? Does PhyX-AF overlap with PhyX?

2. For each evaluation problem, what fraction of the top-5 retrieved memory entries come from the same or a near-duplicate problem? What is Sem after removing those entries?

3. Does the physics semantic judge receive the image? Which model is the judge C inside the training reward, and is it the same as the evaluation judge? What are the judge's FP and FN rates on ConsistencyCheck-1K?

2 comments

Q1: Did the PhyX evaluation problems enter training? Where were the 437 Physlib entries used? Does PhyX-AF overlap with PhyX?

A1: Thanks for your comments. Table 7 mislabels several data sources, and we apologize for the confusion. The actual split, used in all reported experiments, keeps evaluation, SFT, and RL problems disjoint:

DatasetTotalEvalSFTRL
PhyX3,0001,0004501,550
SeePhys2,0001,000661339
PhysUniBench3,3041,0001,1051,199
  • PhyX: “PhyX full test 3,000” is incorrect; none of the 1,000 evaluation problems appears in SFT or RL.
  • 437 entries: these are growing-memory entries distilled from accepted training-set trajectories. They are neither evaluation-derived nor RL samples; “collected from evaluation problems” was a writing error. The corrected RL set has 185,337 examples (3,088 with images).
  • PhyX-AF: all 256 problems (from MMFormalizer) belong to the 1,000 PhyX evaluation problems and have zero overlap with SFT/RL.

We will correct Table 7 and state this split explicitly.

Q2: The external comparison does not match the backbone, teacher, tools, or inference budget, and it lacks more physics baselines and MMFormalizer.

A2: Thanks for your comments. We agree that system-level and controlled comparisons should be separated. External baselines use direct generation without tools or memory, so their gap to GroundForm reflects the full system rather than VHC alone. VHC is isolated by internal runs sharing the SFT checkpoint, data, and training settings (avg Sem, %):

MethodMathPhysics
GRPO73.749.9
Shuffled VHC72.949.2
VHC78.254.6
GroundForm80.858.2

Same tool protocol. With GroundForm's tool interface and call budget, Gemini-2.5-Pro's average math Sem rises from 10.1% to 23.6% (miniF2F 36.9, ProofNet 21.9, Putnam 20.5, AIME25 15.0).

More physics baselines (direct generation, Sem %):

ModelPhyXSeePhysPhysUniPhyX-AFAvg
GPT-565.049.040.035.047.3
Claude-3.7-Sonnet50.036.028.023.034.3
Gemini-2.5-Pro58.143.832.925.440.1
Gemini-3-Pro66.053.042.038.049.8
Qwen3-VL-235B48.035.027.022.033.0
Qwen2.5-VL-72B37.026.020.016.024.8
GroundForm78.151.748.254.758.2

Gemini-3-Pro is higher on SeePhys (53.0 vs. 51.7); GroundForm's average Sem exceeds the strongest baseline by 8.4 points, and its average Syn (92.2) exceeds all baselines (at most 77.5).

MMFormalizer on PhyX-AF (Syn / Sem, %):

Underlying modelMMFormalizer
GPT-541.1 / 31.9
Gemini-3-Pro50.7 / 44.9
Gemini-2.5-Pro24.6 / 19.9
Qwen3-VL-235B10.7 / 8.4
Qwen2.5-VL-72B2.9 / 0.7
Qwen3.5-9B87.0 / 15.0
GroundForm (Qwen3.5-9B)95.7 / 54.7

Qwen3-8B backbone (math avg Syn / Sem, %): base 13.3 / 10.2, SFT 61.8 / 51.0, GRPO 78.3 / 68.3, GroundForm-8B 84.3 / 76.0, ReForm-8B [1] (reported) 85.5 / 64.3. GroundForm-8B exceeds GRPO by 7.7 Sem points. Our SFT stage (51.0%; 57.6% at 9B) is below ReForm-8B, so the gain comes from RL and system components; teachers also differ, so this is not fully controlled.

We will label direct-generation, MMFormalizer, and GroundForm settings separately.

Q3: What fraction of the top-5 memory entries comes from the same or near-duplicate problems? What is Sem after removing them?

A3: Thanks for your comments. The fraction is 0%. Memory is built only from accepted training-set trajectories and frozen at evaluation, and evaluation problems are disjoint from training (A1); shared laws are cross-problem reuse, not near-duplicates. Excluding such entries thus leaves Sem unchanged (58.2%). Removing all memory from the fixed trained policy lowers physics Sem to 54.8% (−3.4).

Q4: Does the physics semantic judge receive the image? Which model is used for training and evaluation? What are the FPR and FNR?

A4: Thanks for your comments. Yes. In both training and evaluation, the judge receives the original problem text, the original image as a separate visual input, and the generated Lean statement. The judge C in the training reward and the evaluation judge are the same model, Qwen3.7-Plus non-thinking, with the same prompt. The manuscript's prompt template omits the image input; we will add it.

On ConsistencyCheck-1K, with human-labeled “semantically consistent” as the positive class (TP = 578, FN = 60, FP = 65, TN = 297):

AccuracyPrecisionRecallF1FPRFNR
87.5%89.9%90.6%90.2%18.0%9.4%

The judge makes both false accepts and false rejects, so Sem approximates human judgment. These rates cover the whole set, not only image-based physics.

[1] ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization

Dear Reviewer,

Thank you again for your time and thoughtful comments on our paper. We have posted a detailed response to your concerns. As the discussion period is coming to an end, we would greatly appreciate it if you could take a look and let us know whether our response has addressed your concerns. We are happy to provide any further clarification if needed.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.