acceptodds
Under review as a conference paper at ICLR 2027

What Feedback Does a Prompt-Optimized Logic Translator Need Under a Proof-Based Judge?

Abstract

Translating natural language into formal logic lets a theorem prover do the inference: reasoning becomes a checkable procedure, and an entailment verdict comes with a proof. What such a translator needs is rarely measured, because published systems are evaluated as packages, each under its own data, models, and judge. We hold the judge fixed, one proof-based judge for every condition, and vary the feedback a prompt-optimized LLM translator receives, its weights frozen. Several models are evaluated on lexical, structural, and monotonicity entailment datasets. We compare feedback sources (target formulas from a symbolic pipeline or entailment labels with the prover's diagnostics), feedback channels, delegation to notation or code, and sentence-level versus pair-level translation. With a strong reflection LM, label feedback fixes the conventions the training pool penalizes, but leaves the two calls disagreeing on argument structure and on predicate names: under a matched training pool, the label-trained translator trails the formula-trained one by eight points on lexical entailment. The condition that shows the formulas as text in the feedback closes that gap to the one that scores against them. A code layer or a one-page notation in the seed prompt each make up a large part of the shortfall without the formulas. Applied to the pair instead, the same prover feedback scores highest on lexical entailment and lowest on the non-entailment list; but on lexical entailment two of every five pairs it proves are merged, the hypothesis formula mapping into the premise formula although the hypothesis sentence has a word the premise lacks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
neuro-symbolic reasoning, autoformalization, natural language inference, theorem proving, prompt optimization, formal semantics, semantic parsing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6pfaOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper provides a profound investigation on influential factor of Prompt-Optimized Logic Translator. It has defined 4 factors: supervision, channel, notation / adaptor and translator unit. The motivations are clear that different paper has their own judger. And the experiments are valid to prove the claims.

Critical strengths
  1. The topic, and the claims provide insights to future research.
  2. The experiments settings are controllable and valid to support the claims.
  3. Clear failures and prompts are shown to provide reproducibility.
Critical weaknesses
  1. The presentation is a bit hard to follow. The narratives needs to be fixed for example the academic writing is hard to follow - some of them are too long and hard to comprehend such as line20-23 and may terms need to be introduced such as P=H、arity conflict、name variant- but it is minor.
  2. Section 4 lacks one comparison: you miss the reward only group
  3. (minor and maybe future work) 4 influential factors are separatedly investigated but what about their joint effects?
  4. Whether the judger is biased? - It is now Rocq + nltac while it has similar notation from silver settings.
Questions for authors

Please see above. I found this paper up to the standard of the acceptance threshold. I may reconsider the overall assessment during the discussion period once the author clarify my concerns.

Reviewer_5LsnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies which feedback improves a frozen LLM’s natural-language-to-logic translation through prompt optimization under a shared proof-based judge. Experiments across several model and benchmarks show that reference formulas provided as textual feedback achieve performance comparable to formula-equivalence scoring.

Critical strengths
  1. From my understanding, the comparisons on the same training data make the role of feedback clearer.

  2. The paper tests simple fixes, such as a short guide for writing formulas, which makes the results practical and easy to follow for ways to improve the translator.

  3. Whether the formulas preserve the original meaning is additionally chekced. This helps explain why a valid proof does not always mean the translation is correct.

Critical weaknesses

From my understanding, the label-based method gets full reward for answering “unknown” on SICK contradiction cases. The same answer is counted as wrong at test time. This may partly explain its lower score than formula-based training.

  1. Evaluation coverage. The main HANS test set contains only positive cases, and the SICK evaluation excludes neutral cases, which raises some concern about whether the results hold on balanced data and on the full test sets.

  2. Translation quality. Since reference formulas are selected based on correct final answers, they may not always preserve the meaning of the original sentences, and the "merge" rate could also flag valid paraphrases. Some manual check or human validation shall be beneficial and essential.

  3. Clarity of presentation. With many experimental settings, it can be hard to keep track of the feedback each model receives. A pipeline figure and a single worked example followed across settings would make the setup and comparisons much easier to follow.

Questions for authors

Please refer to the weaknesses section. I believe the paper would benefit considerably from a revision focused on readability.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5Lsn

Thank you for the review. We answer your points in order. A short overview of the paper (pipeline, the feedback each condition receives, one example) is in our general response at the top of the forum.

1. Training reward vs. evaluation scoring. You are right. On a SICK pair labeled contradiction, the label reward used during optimization gives full credit to an unknown verdict as well as to a proved contradiction (lenient); the evaluation scoring gives credit only to the proved contradiction (strict). Why the reward was lenient: SICK's contradiction labels assume that the two sentences describe one scene, so the set contains pairs whose negation cannot be proved, and on those pairs an unknown verdict is not an error. Why the evaluation is strict: we chose the reading that does not credit silence, so that the SICK numbers are not inflated by contradictions nobody proved. Scored by the lenient score, the gap between S and the control on the matched pool shrinks from .080 to .046 but does not close (.127 to .074 on the full pool of §3). So the mismatch explains part of the gap, as you suggest, and not all of it. We will keep strict as the headline reading, report the lenient reading for SICK beside it, and give the gap under both.

2. Evaluation coverage (W1). The SICK list drops the neutral pairs, and the HANS list holds only entailment pairs, and you ask whether the results hold on balanced data and on the full test sets. HANS: non-entailment is evaluated on a separate list of 570 pairs whose gold is unknown (§2; table in the appendix). On a rebuilt version of that list, balanced over its fifteen constructions, the share of pairs wrongly proved is .24 for the control (.24 with the adapter), .18 for silver-view, .18 for S, and .16 for notation. On average, silver-view, S, and notation prove fewer wrong pairs than the control. The adapter gains seven points on the entailment list, while wrong proofs on this list increase by only two to four out of 570. The complete table will go next to Table 3. SICK: the neutral pairs are not evaluated, because the neutral class is not clean. For example, "A boy is standing outside the water" / "A boy is standing in the water" is labeled neutral, although its label in the B-to-A order is contradiction. In SICK's own directional labels, 319 of the 2,790 neutral test pairs (11%) carry a contradiction judgment in one direction, so a proof on a neutral pair is not always an error. We will state this as a limitation.

3. Translation quality (W2). We agree on both points. The silver formulas come from a CCG parser with hand-written templates. We keep a formula only where that pipeline proves the gold label. This does not guarantee that the formula preserves the meaning. What we can report is that the pipeline rarely proves something wrong (0 false decisive verdicts on SICK and 2 on MED, Appendix B). The merge rate can also flag valid paraphrases, and §5 says so: it is a diagnostic, not a direct measure of unfaithfulness. On pairs whose gold label is entailment, we cannot separate the two automatically. Where the gold label allows a check, the same rewriting produces wrong proofs: on the HANS non-entailment list no proof can be valid, and refinement still turns 59 pairs into proofs; on MED its wrong decisive verdicts rise from 67 pairs to 146 pairs (§5). We will check a sample of 100 merged pairs by hand and report the result.

4. Clarity (W3). We agree. We will add a pipeline figure with one worked example, and a list of what each condition receives as feedback, as in the general response.

5. Training data and presentation (§3). §3 compared L and S on different training pools, so the gap between them mixes the feedback with the data. To optimize against silver formulas, we first run the symbolic parser on every pair and keep only the pairs where it proves the dataset's own label. Every pair has a label, so L used all 16,734 pairs; 3,117 pairs remained for S. We included this comparison to ask whether label-only optimization can learn on the pairs the pipeline cannot handle. But it makes the paper harder to read. The updated paper will start from the matched-pool comparison, which you name as a strength; the full-pool table will move to the appendix, with no number removed.

We will post the updated paper with a list of changes.

Reviewer_rjVBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the role of prompts and verifier feedback in the task of autoformalization to better understand what factors can yield to success in such NL->formal logic translation. To do so, the authors used different translators produced using GEPA and use a reflection LM to provide a score, feedback to propose a revised instruction. Candidates survive by their score on a held-out validation set. Task models have reasoning turned off and instead chain-of-thought (CoT) is used.

The setup is one judge with six different conditions. The silver arm scores formulae by prover-equivalence wherein the score is a linear combination of well-formedness, and other criteria. The label arm is label-only and returns a ternary reward: 1.0 when the verdict equals the label, 0.2 for a wrong label and 0 for a parse failure. The bare arm removes WordNet abduction steps from reward and juding.

The evaluation is performed on 3 test datasets. In section 4, the authors evaluate how much feedback the silver formula adds. Next, the authors check if providing more than one sentence to the translator can yield meaningful improvement. The authors then conclude the study with ablations on the optimizers, the reflection LM and others.

Critical strengths

The motivation is interesting and certainly valuable but I am unable to dissect the key points

Critical weaknesses

As a reader who has experience in formal logic, this paper is not accessible to the AI community at all. It is very dense and very difficult to read. The key contributions are not neatly placed and are lost

Also, some of the results are not really surprising. Getting feedback signals from a verifier is well-known and intuitive.

Finally, this paper is all about prompt optimization and the like. However, not a single prompt is surfaced in the paper. This is the core learning here and it is not provided and as a result, reproducibility of these results will be hard as well i think.

Questions for authors

I do not think this paper is ready for publication. I'd encourage the authors to make the paper easier to read and distill the key concepts. Introducing an example early on in the paper might aid its readability.

Im happy to revise my score if i have misunderstood the work.

1 comment
AuthorsOpenReview ↗

Response to Reviewer rjVB

Thank you for the review. You asked for the key concepts in distilled form and for an early example; we posted both as a general response at the top of the forum (TL;DR). Below we answer your points.

1. Readability. We agree that the paper is too dense and that its key points are not placed where a reader finds them. The general response is how we will present it: the question, the pipeline, one example, what each condition receives, and the takeaway. One structural change is described in point 4.

2. "Some of the results are not really surprising." We agree that feedback from a verifier is known to help, and the paper does not present that as a finding. Its question is which information in the feedback the translator needs, with the prover held fixed. The answers are the ones in the takeaway of the general response. Label feedback and prover diagnostics leave the two separately translated formulas disagreeing on argument structure and predicate names. Showing the silver formula as text closes that gap, and a one-page notation guide or a small code adapter recovers much of it without any silver formula. Translating the pair together scores highest on SICK, but two of every five pairs it proves are merged. And none of the feedback teaches lexical knowledge.

3. No prompt is shown. You are right that no optimized prompt is printed. Each one is three to five pages long, and we left them out for length; we apologize for that. The paper analyzes the prompts in §6, with a table of the rules each prompt states in Appendix D, and it prints the neutral seed prompt in Appendix A. We will add the optimized L prompt (Qwen-235B, seed 0) in full to the appendix, and all optimized prompts will be released with the code. As an example, this is what that prompt says about the construction in the example of the general response:

- IMPORTANT ARITY RULE: give a verb predicate the SAME arity a paired
  sentence would use. If the sentence has an explicit object, model it as a
  binary predicate. Do not silently drop objects, because the matching
  sentence will keep them (e.g. _write(x1,x2) must not be reduced to
  _write(x1) when there is/was an object). Represent the verb with the
  arity implied by its stated participants.

The rule reads the number of arguments off each sentence, so "paid the lawyer" gets two and "paid" gets one, and that pair fails.

4. Training data and presentation (§3). §3 compared L and S on different training pools, so the gap between them mixes the feedback with the data. To optimize against silver formulas, we first run the symbolic parser on every pair and keep only the pairs where it proves the dataset's own label. Every pair has a label, so L used all 16,734 pairs; 3,117 pairs remained for S. We included this comparison to ask whether label-only optimization can learn on the pairs the pipeline cannot handle. But it makes the paper harder to read. The updated paper will start from the matched-pool comparison; the full-pool table will move to the appendix, with no number removed.

We will post the updated paper with a list of changes.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: the paper in short (TL;DR)

We thank the reviewers. Here is the paper in short.

Motivation. In many systems, LLMs are used to translate sentences into logical expressions, and provers are used to perform reasoning. The nature of the information provided as feedback to the translator has not been studied in isolation. We conduct optimization experiments under multiple settings that differ only in the feedback provided.

The pipeline in five steps.

  1. Translation. A frozen LLM translates a sentence into a logic formula. The prompt is the only thing that changes during optimization. The premise and the hypothesis are translated in separate calls.
  2. Proof. A Rocq prover tries to prove the hypothesis formula (H) from the premise formula (P). It can add WordNet axioms (a man is a person). The result is a verdict: yes (P → H), no (P → ¬H), or unknown.
  3. Feedback. Each training pair yields a score and a feedback text, which differs by condition (listed below).
  4. Update. A reflection LM reads a minibatch of these results and rewrites the prompt. Candidates are kept by their score on a validation pool (GEPA).
  5. Evaluation. The optimized prompt is run on the evaluation lists, and every condition is scored by the same prover and the same rule.

We optimize a prompt because the prover's feedback accumulates into it: the prompt can be reused for every new sentence, read to see what was learned, and carried to a new domain.

Formula example.

premise     The actor paid the lawyer.  ->  exists x1 x2. (_actor(x1) & _lawyer(x2) & _pay(x1, x2))
hypothesis  The actor paid.             ->  exists x1. (_actor(x1) & _pay(x1))
prover      unknown   (gold: entailment)

Both formulas are fine on their own, but the pair fails because _pay has two arguments in the premise and one in the hypothesis. The silver formulas avoid the problem by writing the verb as a predicate on an event, with each argument as its own conjunct:

premise     exists x1 x2 e. (_actor(x1) & _lawyer(x2) & _pay(e) & Subj(e) = x1 & Acc(e) = x2)
hypothesis  exists x1 e.    (_actor(x1) & _pay(e) & Subj(e) = x1)

Now the hypothesis is the premise with one conjunct removed, so the proof goes through.

What the feedback can contain. The feedback is a text created for each training condition and is the input of the LLM that rewrites the prompt.

  1. Output formulas. The formulas translator produced for natural language sentence.
  2. Verdict. The prover's result on the two formulas: yes, no, or unknown, next to the gold label.
  3. Diagnostics. Three items taken from the prover: the subgoal it could not close (example: _person(x1)); the predicate names in the hypothesis that do not appear in the premise formula (example: _person); the WordNet axioms added during proof retries (example: forall x. (_man(x) -> _person(x))).
  4. Silver formula. The formula from the symbolic pipeline
  5. Score. A number from 0 to 1 used to keep or drop prompt candidates. For condition L it reflects whether the verdict matches the gold label; for condition S, it reflects how close the formula matches the silver formula.

The conditions (§2, Table 1).

  • L (label arm): the two formulas (1), the verdict (2), the diagnostics (3), and the label score (5).
  • L-bare: same as L, but without WordNet axioms.
  • S (silver arm): output formula (1), the prover's result (2), the diagnostics (3), the silver formula (4), and the silver score (5). Here 2 and 3 are computed between output formula and the silver formula.
  • silver-view: same as L (1–3, 5), plus the silver formulas (4) shown on failed pairs. The score stays the label score; the silver formula is shown, not scored (§4).
  • notation: same as L; the difference is a one-page notation guide in the seed prompt (§4).
  • adapter: same as L; the difference is that the formulas pass through a small code adapter before the prover (§4).
  • refinement: the pair translated together at evaluation time (§5).

Takeaway. With a prover doing the inference, the hard part is making two separate translations agree.

  • Under the condition L the prompt learns what the training pool penalizes, but the two separately translated formulas still disagree on argument structure and predicate names (SICK: L .415 in the control row, S .495; §4, Table 3).
  • Under the silver-view condition, showing the silver formula as text closes the gap. The notation guide and the code adapter recover much of it without any silver formula (§4, Table 3).
  • Translating the pair together (refinement) scores highest on SICK (§5, Table 4), but two of every five pairs it proves are merged: the hypothesis formula collapses into the premise formula although the hypothesis has a word the premise lacks.

What the feedback does not teach is lexical knowledge: the prompt learns conventions, and word-level relations have to come from a lexical resource or from weights.

AuthorsOpenReview ↗

List of changes

Updated paper: list of changes

We have uploaded the updated paper. Numbers below refer to the updated paper; our earlier replies used the numbers of the submitted version, and the mapping is at the end.

Changes we promised

  1. Order of the main text (6pfa, 5Lsn, rjVB). The main text now starts from the comparison on one training pool (§3). The full-pool results (submitted §3 and its table) are in Appendix G. No number was removed.
  2. Strict and lenient scoring (6pfa, 5Lsn). Strict accuracy remains the headline. §2 states where the label reward and the evaluation differ on SICK. Table 2 adds a SICK lenient column. The gap between S and the control on SICK is 8.0 points strict and 4.6 points lenient (§3); on the full pool it is 12.7 and 7.4 (Appendix G).
  3. 2×2 framing (6pfa). Removed. §3 explains why we do not run a condition that scores against the silver formulas without showing them.
  4. Pipeline figure and example (6pfa, 5Lsn, rjVB). Figure 1 shows the pipeline, the optimization loop, and one test pair (The actor paid the lawyer / The actor paid) under two conditions.
  5. What each condition receives (5Lsn). §2 lists the five elements of the feedback, and Table 1 marks which of them each condition receives.
  6. Order of presentation (rjVB). The introduction now gives the question, the pipeline, the example, and the findings in that order; §2 then describes the conditions.
  7. Abstract and terms (6pfa). The abstract is shorter. Arity conflict and predicate-name variant are defined in §2 (Terms); merged and P=H are defined in §4, where they are first used.
  8. HANS non-entailment list (6pfa, 5Lsn). The list is rebuilt: 570 pairs, 38 from each of the fifteen non-entailment subcases, disjoint from every training pool. Table 2 reports it as a column, and Table 5 (Appendix A.6) reports every condition.
  9. Limitations (6pfa, 5Lsn). Added: the factors are not fully crossed, and SICK's neutral pairs are not evaluated (319 of the 2,790 neutral test pairs carry a contradiction label in one direction).
  10. Hand check of 100 merged pairs (5Lsn). Appendix A.4 gives the criteria and the result, and §4 summarizes it. The sample is 25 pairs from each refinement run on SICK and on MED, among the pairs that refinement revised. The premise supports the rewrite in 77 pairs. It does not in 4 of the 50 SICK pairs and in 18 of the 50 MED pairs.
  11. An optimized prompt in full (rjVB). Appendix H prints the optimized L prompt (235B, seed 0). All optimized prompts will be released with the code.

Other changes

  • Baseline of §3 and §5. The comparisons in §3 and the error analysis in §5 now use the control (label reward on the silver pool) as the baseline, so that every comparison in §3 is on one pool. L (label reward on the full pool) is the starting point of §4 and of the prompt reading in §5, and its results are in Appendix G. §6 states that its L and S runs were optimized on different pools.
  • Wording. We rewrote the introduction, §2 to §4, and the conclusion in simpler sentences, and we use the condition names of Table 1 in the text and in every table.

Numbers that changed

  • Non-entailment. All numbers on the non-entailment list are recomputed on the new list (for example, L .758 → .782). The conclusions are the same.
  • Merged shares. We recomputed them with the script that will be released (analyze_merged.py). SICK: L 8% → 7%, S 5% → 4%, refinement 43% → 43%, refinement from L 41% → 40%. MED (L / refinement / refinement from L): 14% / 33% / 28% → 15% / 32% / 29%.
  • Reference condition. Where §3 and §5 gave a number for L, they now give it for the control (for example, arity conflicts are 53–73% of the control's HANS errors).

Numbering (submitted → updated)

SubmittedUpdated
§3 (full pool)Appendix G
§4 (matched pool)§3
§5 (pair)§4
§6 (prompts)§5
§7 (recipe)§6
Table 3Table 2
Table 4Table 3
Table 6Table 5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.