- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
GroundForm is a tool-using agent on Qwen3.5-9B that turns textual math problems and image-plus-text physics problems into Lean 4 statements.
While formalizing, it can search Mathlib, a physics library, two growing memories and the web.
Training is SFT on trajectories distilled from strong closed models, then GRPO with Verified Hindsight Credit (VHC).
In VHC, statements in a rollout group that compile and pass an LLM judge become targets, and a frozen reference model scores how much each tool call raises the likelihood of those targets, so every retrieval step gets its own credit. The two memories store verified math and physics experience during training and are used at test time.
The evaluation covers four math and four physics benchmarks, reporting 80.8% math and 58.2% physics semantic accuracy, a shared-backbone ladder (GRPO, shuffled VHC, VHC, full model), and analyses of tools, memory and images. VHC is a neat way to get turn-level credit without gold formalizations, though it is close to recent information-gain style credit. The physics results have data and judge problems that need fixing before the numbers can be trusted.
Critical strengths
-
VHC is a simple and well-fitted answer to a real problem: how to credit intermediate retrieval steps when there's no gold formalization. The shared-backbone ladder is well designed, and the shuffled-credit control falls below GRPO while real VHC gains about 4.5 points in math. This matters because it suggests that where the credit lands matters, not just having a denser signal, and other agent RL work could reuse this.
-
The image analysis gives a clear lesson for multimodal formalization. Success depends heavily on whether what the image shows is actually written into the Lean statement, and masking the image cuts image-necessary accuracy sharply. This is useful to anyone building multimodal autoformalizers, independent of the training recipe.
Critical weaknesses
-
The physics evaluation looks contaminated. Table 7 puts 'PhyX full test' (3,000 items), 'Physlib entries collected from evaluation problems' and 'PhysicsEval (train + test)' in the RL pool, while 1,000 PhyX problems are used for evaluation, and I could not find any decontamination step. The memories are built from accepted training trajectories and used at test time. With static libraries only, the full model scores about the same as VHC (Table 9), so the whole memory gain comes from test-time retrieval, which is exactly where lookup of test items would show up. This matters because it affects the 58.2% physics number, the memory and transfer claims and the PhyX-AF efficiency result. Fixing it needs retraining and re-evaluation on clean data, not just a clarification.
-
The physics metric isn't shown to measure faithful formalization, and the comparisons and positioning are incomplete. The physics judge is a closed LLM with a lenient rule, it is validated only on math, and accepted outputs can be near-tautological: the standing wave statement in App. G assumes d = 1/2 * (v/f) as a hypothesis and concludes v = f * 2 * d. Math baselines are copied single-pass ReForm numbers (with ReForm-32B left out), physics has only Gemini-2.5-Pro, and the SFT teachers are never evaluated with the same tools. VHC is also close to IGPO (turn-level information gain, ICLR 2026) and likelihood rewards like RLPR, and retrieval-grounded autoformalizers (RAutoformalizer, DRIFT, Aria) aren't cited. So both the size of the gains and the novelty are less clear than the abstract suggests.
Questions for authors
- Are the 1,000 evaluated PhyX problems, and the PhyX-sourced items of PhyX-AF, inside 'PhyX full test' or 'PhyX-100'? What exactly are the 437 'Physlib entries collected from evaluation problems', and can any memory or PhysSearch entry come from an evaluation item? Please report physics results after removing evaluation items from the RL data, memories and search index, with web search blocked from benchmark sources. This is the main thing that could change my score.
2 comments
Dear Reviewer,
Thank you again for your time and thoughtful comments on our paper. We have posted a detailed response to your concerns. As the discussion period is coming to an end, we would greatly appreciate it if you could take a look and let us know whether our response has addressed your concerns. We are happy to provide any further clarification if needed.
Best regards, Authors
A1: Thanks for your comments. Table 7's “PhyX full test 3,000” label is incorrect. The actual split, used in all reported experiments, keeps evaluation, SFT, and RL problems disjoint:
The 256 PhyX-AF problems are a subset of the 1,000 PhyX evaluation problems and were excluded from SFT and RL. The 437 entries are growing-memory entries distilled from accepted training-set trajectories, not evaluation-derived content, RL samples, or PhysSearch index entries. Neither the frozen evaluation memory nor the PhysSearch index contains evaluation problems or derived content, and benchmark sources are blocked in WebSearch during evaluation. The reported physics results were obtained under exactly this setting:
We will correct Table 7 and document the isolation of each component.
A2: Thanks for your comments.
Sem and faithfulness. Following ReForm [1], we use model-judged Sem as a proxy for faithful formalization; it is not equivalent to formal verification of faithfulness. For image-based physics, we validated the judge against human labels on 240 samples (40 original + 200 new) with 82.1% accuracy (72.5% / 84.0%). In training and evaluation, the judge receives the original text, image, and Lean statement. We agree with the reviewer on the Appendix G standing-wave example: once the node-spacing relation is stated as a premise, the conclusion is its algebraic rewriting, and the judge accepted this weak formalization. We will report it as a limitation, temper our Sem claims, and study how to detect such cases.
Tool protocol. To measure how much tools alone help an external model, we gave Gemini-2.5-Pro the same tool interface and call budget as GroundForm: its average math Sem rises from 10.1% to 23.6% (miniF2F 36.9, ProofNet 21.9, Putnam 20.5, AIME25 15.0). The training contribution of VHC is isolated by internal runs sharing the backbone, SFT initialization, and training configuration: VHC improves over GRPO by 4.5 (math) and 4.7 (physics) Sem points, while shuffling credit across tool turns lowers performance below GRPO.
More baselines. Physics, direct generation, avg Syn / Sem (%): GPT-5 75.3 / 47.3; Claude-3.7-Sonnet 62.0 / 34.3; Gemini-2.5-Pro 61.1 / 40.1; Gemini-3-Pro 77.5 / 49.8; Qwen3-VL-235B 66.8 / 33.0; Qwen2.5-VL-72B 55.0 / 24.8; GroundForm 92.2 / 58.2. On PhyX-AF, MMFormalizer obtains 41.1 / 31.9 (GPT-5), 50.7 / 44.9 (Gemini-3-Pro), 24.6 / 19.9 (Gemini-2.5-Pro), 10.7 / 8.4 (Qwen3-VL-235B), 2.9 / 0.7 (Qwen2.5-VL-72B), and 87.0 / 15.0 with our Qwen3.5-9B backbone, versus 95.7 / 54.7 for GroundForm.
Qwen3-8B backbone (math avg Syn / Sem, %): SFT 61.8 / 51.0, GRPO 78.3 / 68.3, GroundForm-8B 84.3 / 76.0, ReForm-8B (reported) 85.5 / 64.3. Our SFT stage alone is below ReForm-8B; the gain comes from subsequent training and system components. Our SFT trajectories come from Claude Opus 5 and Qwen3.7-Plus, whereas ReForm uses a Qwen3-235B-A22B-Thinking multi-agent pipeline, so this is not a fully controlled comparison. We did not reproduce ReForm-32B and will qualify cross-model claims accordingly.
Related work and VHC. IGPO rewards turns by the change in the probability of the correct answer, and RLPR uses the reference answer's token probability as the reward. RAutoformalizer, DRIFT (decomposition with premise retrieval), and Aria (dependency-graph-guided retrieval) use retrieval to aid Lean formalization. VHC shares likelihood-based credit with the former and retrieval-based grounding with the latter. Its specific contribution is a turn-level likelihood-potential difference without a unique gold formalization: candidates in a rollout group that pass Lean compilation and the semantic judge serve as targets, a frozen SFT reference model scores their aggregated likelihood before and after each tool turn, and the difference credits that turn. VHC thus needs no human gold statement but relies on the compiler and the judge. We will add this discussion and narrow our novelty claims accordingly.
[1] ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization