- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
ReExp turns prior autoformalization corrections into typed, anchored repair experiences, admits them using independent groups, and freezes the resulting library before test-time use. A repair is applied transactionally at an eligible state, with rollback on rejection, before the shared refinement loop resumes. Across five benchmarks and four backbones, it reports 6.78–14.67 percentage-point gains over the Logic-LM baseline and smaller gains over a retrieval comparator. A blinded ProverQA audit also examines semantic faithfulness. The executable reuse mechanism is promising, but the relationship between committed repairs and reported inference resources needs reconciliation.
Critical strengths
-
Reusing corrections as executable, role-typed edits rather than only retrieved text is a concrete methodological contribution. Independent admission, a frozen library, and transactional rollback create a clear path for transferring an observed correction while limiting test-time adaptation (Sections 3–4; Appendices A–C).
-
The evaluation covers 20 backbone/benchmark settings, separates source/admission/selection/test groups, and reports paired uncertainty. The semantic audit goes beyond answer accuracy: blinded annotators assess paired ProverQA representations, with high agreement and reported faithfulness improvements. Although conditional on the audited population, this provides useful evidence about a common failure mode of autoformalization.
Critical weaknesses
-
Tables 6 and 8 appear inconsistent with the stated shared-loop protocol. For Qwen on Deduction, Table 6 reports 18 committed repairs over 300 cases, with four rejections. Appendix C.1 states that noncommitted trajectories match the baseline, yet Table 8 reports 3.51 versus 1.04 mean LLM calls: approximately 741 additional calls. With the shared initial call, at most three refinements, and a fallback, even allowing all 22 attempted edits to change a five-call trajectory cannot explain this difference. Likewise, the reported output-token increase is about 3.96 million tokens, exceeding a generous 22×5×8,192-token bound under the stated cap. This may reflect different cohorts or accounting definitions, but until reconciled it undermines the reported resource comparison and understanding of the executed algorithm.
-
The causal case for experience reuse needs a clearer cost-matched comparison. The reported benefit over the retrieval comparator is modest, while offline extraction/admission and test-time call/token costs are material. The current accounting issue prevents assessing whether the additional quality comes from transferable edits or a different effective refinement budget. The faithfulness audit is also restricted to 451 ProverQA commits with both representations, so it supports that conditional population rather than all interventions or datasets.
Questions for authors
-
Please reconcile the Qwen Deduction committed/rejected counts in Table 6 with Table 8's call and output-token means. Are the cohorts, denominators, or definition of an LLM call different? Per-case trajectories for committed, rejected, and untouched cases would resolve this concern.
-
Once accounting is reconciled, does ReExp retain its advantage over retrieval and shared refinement at matched total inference calls/tokens, including fallback? This would materially affect attribution of the improvement.
-
For the faithfulness audit, how do the excluded commits and untouched cases differ from the 451 paired cases? Please delimit the conclusion to the population actually audited.