acceptodds
Under review as a conference paper at ICLR 2027

ReExp: From Autoformalization Corrections to Transferable Repair Experience

Abstract

Autoformalization enables solver-based reasoning by translating natural-language problems into executable symbolic representations, while feedback-driven refinement can correct faulty formalizations. Such refinement also produces correction histories, but a successful correction does not by itself reveal which local changes can be safely reused on new problems. We introduce ReExp, a framework that transforms failure-to-success correction histories into transferable repair experience. ReExp replays recorded corrections to isolate sufficient edits, abstracts them into typed and rebindable repair programs, and retains only repairs that demonstrate positive utility on held-out states. At inference, selected repairs are rebound and executed on compatible formalizations before standard refinement resumes. Across five logical reasoning benchmarks and four backbones, ReExp improves over within-instance solver-feedback refinement in all 20 settings, with per-backbone mean gains of 6.78–14.67 percentage points. It also outperforms retrieval-based experience reuse in most settings, with corresponding improvements in semantic faithfulness.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-Symbolic Reasoning, Autoformalization, Symbolic Program Repair

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ViDVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ReExp turns prior autoformalization corrections into typed, anchored repair experiences, admits them using independent groups, and freezes the resulting library before test-time use. A repair is applied transactionally at an eligible state, with rollback on rejection, before the shared refinement loop resumes. Across five benchmarks and four backbones, it reports 6.78–14.67 percentage-point gains over the Logic-LM baseline and smaller gains over a retrieval comparator. A blinded ProverQA audit also examines semantic faithfulness. The executable reuse mechanism is promising, but the relationship between committed repairs and reported inference resources needs reconciliation.

Critical strengths
  1. Reusing corrections as executable, role-typed edits rather than only retrieved text is a concrete methodological contribution. Independent admission, a frozen library, and transactional rollback create a clear path for transferring an observed correction while limiting test-time adaptation (Sections 3–4; Appendices A–C).

  2. The evaluation covers 20 backbone/benchmark settings, separates source/admission/selection/test groups, and reports paired uncertainty. The semantic audit goes beyond answer accuracy: blinded annotators assess paired ProverQA representations, with high agreement and reported faithfulness improvements. Although conditional on the audited population, this provides useful evidence about a common failure mode of autoformalization.

Critical weaknesses
  1. Tables 6 and 8 appear inconsistent with the stated shared-loop protocol. For Qwen on Deduction, Table 6 reports 18 committed repairs over 300 cases, with four rejections. Appendix C.1 states that noncommitted trajectories match the baseline, yet Table 8 reports 3.51 versus 1.04 mean LLM calls: approximately 741 additional calls. With the shared initial call, at most three refinements, and a fallback, even allowing all 22 attempted edits to change a five-call trajectory cannot explain this difference. Likewise, the reported output-token increase is about 3.96 million tokens, exceeding a generous 22×5×8,192-token bound under the stated cap. This may reflect different cohorts or accounting definitions, but until reconciled it undermines the reported resource comparison and understanding of the executed algorithm.

  2. The causal case for experience reuse needs a clearer cost-matched comparison. The reported benefit over the retrieval comparator is modest, while offline extraction/admission and test-time call/token costs are material. The current accounting issue prevents assessing whether the additional quality comes from transferable edits or a different effective refinement budget. The faithfulness audit is also restricted to 451 ProverQA commits with both representations, so it supports that conditional population rather than all interventions or datasets.

Questions for authors
  1. Please reconcile the Qwen Deduction committed/rejected counts in Table 6 with Table 8's call and output-token means. Are the cohorts, denominators, or definition of an LLM call different? Per-case trajectories for committed, rejected, and untouched cases would resolve this concern.

  2. Once accounting is reconciled, does ReExp retain its advantage over retrieval and shared refinement at matched total inference calls/tokens, including fallback? This would materially affect attribution of the improvement.

  3. For the faithfulness audit, how do the excluded commits and untouched cases differ from the 451 paired cases? Please delimit the conclusion to the population actually audited.

Reviewer_NS6MOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ReExp, a method designed to correct errors that occur when large language models translate natural language problems into logical expressions. It extracts repair rules from past error-correction records and, after a filtering process, applies them to new problems. The paper reports that this approach improves accuracy across five logical reasoning datasets.

Critical strengths

S1. The authors used distill historical error corrections into actionable rules to facilitate their reuse in new tasks. S2.Also they used independent data to verify whether these fixes yield additional benefits, ensuring a clear basis for selection.

Critical weaknesses

W1. Paper is hard to follow. (a) Figure 3 In subfigure (a), aa may mean the seven items; is PP the variable or just the item denoted by PP? What is the pp? What are the L @ # 1 and R @ # 4 denote? Subfigures (b) and (c) have the same problem. It would be better to explain each variable; an explanation table would help the reviewer to understand your details better. (b) Unclear generation and semantic validation of repair preconditions Please explain in detail how the editing preconditions for the repair procedure are generated from the historical error correction records. Taking the R operation in Figure 3 as an example, what conditions does the system use to change (q(H)>q(S)) in the new question to (q(H)<q(S))? Do these conditions check the corresponding natural language meaning, or only check the diagnostic information and expression structure? Please demonstrate the complete preconditions and matching process for this example, and explain how the system handles situations where the structure matches but the original relation direction is correct. This part would be better placed in the main text. W2. Incomplete baseline comparisons. Table 1 omits several baseline results because no implementation has been released for those benchmarks. This alone does not sufficiently justify the missing comparisons. The authors should adapt and evaluate applicable baselines on these tasks, or explain the specific technical obstacles that prevent a fair comparison. Table 2 have the same problems.

Questions for authors

Q1: Inconsistent results across tables. For Qwen3.8-27B on Logical Deduction, Table 1 reports 96.33% accuracy for Logic-LM and 100% for ReExp on 300 questions, corresponding to 11 and 0 errors, respectively. However, Table 6 reports 13 improvements and 2 degradations. These counts cannot describe the same paired predictions: the baseline has only 11 errors available to correct, and two degradations would leave at least two errors in ReExp’s predictions. Please clarify whether the tables use the same evaluation records and metric, and reconcile these counts with the reported accuracies. Q2: How is the refinement budget controlled in the direct-execution ablation? In Table 2, does the model-generated revision in w/o direct execution consume one of the three refinement rounds? After applying the repair, do this variant and the full method have the same remaining refinement budget?

Reviewer_FQqwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ReExp, a framework that transforms failure-to-success correction histories into transferable repair experience. ReExp shows to improve over within-instance solver-feedback refinement.

Critical strengths
  1. The paper has comprehensive experiments across five logical reasoning benchmarks and four backbones.
Critical weaknesses
  1. The accuracy and repair-outcome counts are inconsistent: for Qwen on 300 Logical Deduction questions, Table 1 implies exactly 11 wrong-to-correct changes (96.33% to 100%) over Logic-LM and zero correct-to-wrong changes, whereas Table 6 reports 13 improvements and two degradations. Although the reported net gain is correct, the individual counts are impossible because only 11 baseline errors exist and ReExp’s reported 100% accuracy permits no final degradations. The authors should reconcile these tables and clarify whether they use the same predictions and the same end-to-end evaluation metric. I have concerns over the soundness and correctness of this paper.
Questions for authors

Are you able to reconcile Table 1 and 6 as I mentioned in the weakness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.