acceptodds
Under review as a conference paper at ICLR 2027

Recoverable Proofs: Reverse-Semantic Preference Training for Logical Reasoning

Abstract

Proof-first prompting encourages large language models to solve logical-reasoning tasks by generating intermediate proofs, rationales, or symbolic-style traces before producing a final answer. However, such traces are not necessarily faithful carriers of the original problem semantics: a model may predict the correct label while omitting key conditions, confusing entities or operators, or relying on shortcuts that are difficult to recover from the trace itself. We call this the proof-trace recoverability gap. We propose Reverse-Semantic Preference Training (RSPT), a train-time plug-in method for proof-first logical reasoning. For each problem, RSPT samples candidate proof traces, extracts deterministic semantic slots from the original problem, constructs corrupted slot alternatives, and uses the frozen backbone to score whether the generated trace makes the gold slots more recoverable than the corrupted ones. This recoverability signal is then used to construct same-question preference pairs for standard SimPO/LoRA adapter training. At inference time, the trained adapter produces one deterministic output under an existing proof-first scaffold, without reward scoring, reranking, majority voting, or an additional verifier. Across FOLIO and ProofWriter, RSPT improves all 9 of 9 complete parser-safe same-scaffold settings under the same backbone, parser, decoding budget, and greedy one-output protocol, with gains up to +5.23 points on FOLIO and +10.78 points on ProofWriter. Under a fixed global Symbolic-aided-CoT deployment scaffold, RSPT-Scaffold@1 reaches 69.93% ± 0.57 on FOLIO and 69.28% ± 2.74 on ProofWriter, outperforming the strongest frozen parser-safe baselines in our unified setting. We separate deterministic adapter inference from analysis-only Reward-rerank@8. Mechanism controls show that recoverability-specific construction is clearest on FOLIO, while ProofWriter also benefits from simpler answer-filtered same-question contrastive supervision. These results support a scoped claim: proof-first models can be improved by training them to preserve recoverable problem semantics, rather than only optimizing final-answer correctness.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
large language models, logical reasoning, proof-first reasoning, preference optimization, reasoning faithfulness, semantic recoverability, LoRA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SVsUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a train-time plug-in method, termed Reverse-Semantic Preference Training (RSPT), for proof-first logical reasoning with proof-trace. It samples candidate traces, extracts deterministic semantic slots, constructs corrupted slot alternatives, and apples the frozen backbone to score the influence of the trace. Experimental results across FOLIO and ProofWriter demonstrates the effectiveness of the proposed method.

Critical strengths

This paper is well organized. The proposed reverse-semantic preference training method with trace selection, semantic slot extraction, and trace evaluation is interesting.

Critical weaknesses
  1. The proposed RSPT scores whether the selected semantic slots can be recovered from a trace. However, a trace which preserve these slots may contain invalid or less information for reasoning. Improving the recoverability does not demonstrate that the generated proofs can be logically correct or the trace can be effective. Therefore, it is recommended to discuss the influence of the trace, the semantic slots, and the semantic recoverability independently in three steps.

  2. The experiments were carried out on only one backbone (Qwen2.5-7B-Instruct) and two typical benchmarks. Ablation studies on the proposed RSPT with different LLMs, different semantic recovering strategies, and different trace selection strategies are recommended. More comparison studies on the state-of-the-art techniques with more benchmarks are also suggested.

Questions for authors

/

2 comments
AuthorsOpenReview ↗

Response to Reviewer SVsU (1/2)

We thank the reviewer for finding the paper well organized and the reverse-semantic preference method interesting. Both suggestions, separating the roles of the trace, the semantic slots, and recoverability, and broadening the evaluation, led to new controlled analyses (Appendix B.12, including the new margin analysis in B.12.11).

W1. Separating trace information, semantic slots, and recoverability-guided selection

We agree that preserving semantic slots does not guarantee logically correct or faithful proofs. Following your suggestion, we now address the three components separately.

(1) Does the score depend on the actual trace?

We compared real frozen-model traces with empty, unrelated, and length-matched unrelated traces, keeping the scorer and the slots fixed (Appendix B.12.6 and B.12.11).

For gold-slot likelihood SgoldS_{\mathrm{gold}}, the score that drives selection, real traces score higher than all three controls on both datasets. Against the length-matched unrelated control, the gain is +0.479 nats/token on FOLIO (95% CI [+0.428, +0.532]) and +0.139 on ProofWriter ([+0.129, +0.148]). Against an empty trace, however, the ProofWriter gain is only +0.012.

For the contrastive margin Mslot=Sgold−ScorrM_{\mathrm{slot}}=S_{\mathrm{gold}}-S_{\mathrm{corr}} (Table 23):

DiagnosticFOLIOProofWriter
Real-trace mean margin−0.0789−0.0220
Positive-margin fraction19.61%23.50%
Real − empty margin [95% CI]+0.0264 [+0.0143, +0.0397]−0.0001 [−0.0022, +0.0020]

Real traces also raise the margin relative to the unrelated and length-matched controls on both datasets. However, the absolute mean margins are negative, and the ProofWriter real-versus-empty difference is effectively zero. The trace therefore clearly shapes SgoldS_{\mathrm{gold}}. The margin, by contrast, is only weakly trace-sensitive and does not reliably separate gold from corrupted slots.

(2) What do the semantic slots contribute?

The slot extractor is deterministic and dataset-specific. It represents targets, entities, predicates, operators, and the available facts and rules (Appendix A.1).

  • Label independence. A metadata-substitution audit (4,808 instances, 14,424 gold-label replacements) confirmed that the scorer inputs do not change when gold-answer metadata is replaced. This checks input independence, not extraction correctness.
  • Where the slots matter. On an input-defined 75-example FOLIO subset where key slots matter, full RSPT reaches 61.33%, compared with 54.67% for random and 53.33% for answer-only pair construction (single runs; Table 12). The advantage is larger than on the full split.
  • Not just surface overlap. SgoldS_{\mathrm{gold}} correlates only moderately with n-gram overlap (Spearman ≈ 0.19–0.28), and a matched 5-gram-overlap selector performs below SgoldS_{\mathrm{gold}} selection in every seed, although the adjusted intervals cross zero.

These analyses separate the slot-based scorer from literal overlap, but they do not independently validate every extracted relation or guarantee proof validity.

(3) Does recoverability-guided ranking improve downstream reasoning?

We ran matched three-seed experiments that fix the training questions, sampled candidates, correctness eligibility, rejected traces, optimization settings, and evaluation protocol; only chosen-trace selection changes. With Qwen2.5-7B-Instruct, FULL RSPT exceeds matched answer-order selection by +4.90 pp on FOLIO (95% CI [+1.50, +7.17]) and +2.11 pp on ProofWriter (95% CI [+0.22, +4.18]), in every seed.

  • Gold-slot likelihood alone. Selecting by SgoldS_{\mathrm{gold}} without the margin retains +2.94 and +1.61 pp, with adjusted intervals that exclude zero. FULL minus NO_MARGIN does not establish an independent benefit from the margin.
  • Same examples. On examples that both adapters answer correctly, FULL traces have higher SgoldS_{\mathrm{gold}} than answer-order traces in every seed (Table 21).

These findings support recoverability-guided training-trace selection, mainly through gold-slot likelihood. They do not establish that the resulting proof steps are formally correct.

AuthorsOpenReview ↗

Response to Reviewer SVsU (2/2)

W2. Additional backbones, recovery strategies, and selection baselines

Different LLM backbones. We independently sampled candidate pools and trained new adapters on Llama-3.1-8B-Instruct, without Llama-specific hyperparameter tuning. FULL exceeds matched answer-order selection by +4.08 pp on FOLIO (95% CI [+0.13, +7.78]) and +2.67 pp on ProofWriter (95% CI [+0.28, +5.08]), a second-model-family replication of the chosen-trace ranking effect (Appendix B.12.9). We also report a seed-sensitive Qwen2.5-14B diagnostic and an earlier single-seed Mistral probe that does not establish reliable transfer (Appendix B.10, B.12.10). We therefore do not claim universal backbone robustness.

Different scoring and trace-selection strategies. Under matched conditions, we compare the following chosen-side rules:

  • FULL;
  • NO_MARGIN, which uses gold-slot likelihood only;
  • answer-order selection;
  • 5-gram-overlap selection.

We also evaluate standard random and answer-order two-sided preference construction. FULL has higher accuracy than both two-sided controls in every Qwen seed on both datasets. The adjusted confidence intervals include zero, so we report directional gains rather than statistically established superiority (Appendix B.12.3–B.12.4). These comparisons separate recoverability scoring from candidate order and lexical overlap. They are not an exhaustive comparison of alternative slot-extraction designs.

Benchmarks and baselines. FOLIO and ProofWriter remain our two primary benchmarks. ProntoQA is a supplementary diagnostic, and there simpler preference controls outperform RSPT. Our prompting baselines (Logic-CoT-, SymbCoT-, and COP-style) are controlled variants under a shared parser and decoding protocol, not full reimplementations of those systems. Broader benchmark coverage, alternative slot representations, stronger full-system baselines, and verified proof correctness remain important extensions, and we state them as limitations.

Summary

The revised experiments separate trace dependence, semantic-slot scoring, and downstream preference-selection effectiveness. The best-supported conclusion is that trace-conditioned gold-slot likelihood provides a useful signal for selecting among answer-correct training traces, with matched positive results across two datasets and two model families. We have narrowed our claims accordingly. Recoverability is a training-time proxy, not a certificate of logical proof correctness, and most of the gain comes from gold-slot likelihood rather than the contrastive margin.

We thank the reviewer again for the constructive suggestions, which helped clarify both the empirical contribution and its boundaries.

Reviewer_UtyiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RSPT, a way to build preference pairs for training LLMs on logical reasoning tasks where the model writes a proof before its answer. For each training problem, the frozen model samples several proof traces. A rule-based extractor turns the problem into "semantic slots" (target, entities, predicates, operators, facts, rules), and corrupted versions of these slots are made by flipping negations, swapping entities and similar edits. The frozen model then scores how likely the gold slots are given only the trace, compared with the best corrupted version. This "recoverability" margin is gated by answer correctness and combined with format and anti-copying terms. It is used to pick the best correct trace and the worst incorrect trace for each question. These pairs train a LoRA adapter with SimPO on Qwen2.5-7B-Instruct, and at test time the adapter gives one greedy output. On FOLIO (204 test examples) and ProofWriter (600), the adapter improves all nine prompt setups tested: by +0.16 to +5.23 points on FOLIO and +6.50 to +10.78 on ProofWriter. Reranking eight samples with the reward nearly matches picking a correct sample whenever one exists. In controls, the full method beats simpler pair constructions on FOLIO, but on ProofWriter random or answer-only pairs do as well or better.

Critical strengths
  1. A simple idea with cheap inference. The reward needs only the frozen model and rule-based slots, and test time uses one greedy output with no reward model or verifier.
  2. Careful, insightful comparisons. Each frozen/+RSPT pair shares the backbone, prompt, parser and decoding. The method is run with three seeds, scored with McNemar tests, and checked with a parse-failure rule. The paper openly reports results that go against it: the ProofWriter controls, the drop in Unknown accuracy at high strength, ProntoQA, and Mistral.
Critical weaknesses
  1. The recoverability score itself is not shown to help. The "9 of 9" result compares a fine-tuned model with an untuned one, so most in-domain preference training would likely give gains too. The fair comparison is with the same pairs built without the score, and there the evidence is weak. On ProofWriter, random pairs (68.83%) and answer-only pairs (68.17%) beat full RSPT (66.11%), and on ProntoQA they also beat it (92.00% and 89.00% vs 88.33%). On FOLIO, full RSPT leads the best control by 5 of 204 examples, but the controls are single runs. Appendix B.6 also reports that even the gain over the base model is not significant (p = 0.20–0.34). Two of the nine "improvements" are within noise: +0.16 ± 1.86 and +0.82 ± 0.57.
  2. The reward check uses the gold answer. Eq. 4 multiplies the reward by whether the trace's answer matches the gold label. Reranking with it is therefore close to picking a correct sample with the answer key, so matching Oracle@8 on ProofWriter (85.50 vs 85.50) is expected and says little about recoverability. A fair test would rerank using the recoverability terms alone.
  3. The contrastive part of the score carries almost no signal. The margin should show that a trace makes the gold slots more recoverable than corrupted ones. Yet even the best ProofWriter model averages a margin of −0.0197 on its correct answers (Appendix B.9), so the strongest corruption is slightly more likely than the gold slots. What separates traces is then mostly S_gold, the plain likelihood of the gold slot text, and this rises whenever a trace restates more of the problem. Slot-only pairs do score below full RSPT. But they also drop the format and copy terms, so that comparison doesn't isolate the margin. The paper never scores the slots with an empty or unrelated trace, so it is unclear how much the score depends on the trace at all. The slot extractor (string matching and negation detection on FOLIO) and the corruptions are also never checked for accuracy.
  4. Rejected traces are chosen by format, not by recoverability. In Eq. 4, the recoverability terms are multiplied by answer correctness. For an incorrect trace, the reward is therefore just the format bonus minus the copy penalty. Picking the lowest-reward incorrect trace thus favours traces that break the answer format or copy the most. The pairs may largely teach the model to produce parseable output, which would also explain why one adapter helps across different prompts. This can't be checked: parse-failure rates are given only as "≤ 5%" for the frozen rows and not at all for the adapted ones.
  5. Faithfulness is never measured directly. The motivation is faithful proofs, but the evidence is only accuracy, the method's own reward and one example. Table 3 shows that the model trained on the reward scores higher on that same reward, which is circular. ProofWriter comes with gold proofs, so proof correctness could have been checked.
  6. Some numbers and choices don't line up.
    • Table 2's "Proof-first" row (FOLIO 61.76 → 66.67, ProofWriter 56.83 → 67.28) differs from Table 9 (62.75 → 66.34, 58.00 → 66.11). Section 4.1 suggests both are the bare proof-first setting.
    • The FOLIO controls are compared with RSPT's best seed (66.67), ProofWriter with the mean (66.11), and Table 12 uses a third value (66.18).
    • No dev split separate from the evaluation sets is described. The ProofWriter learning rate was lowered "to avoid over-sharpening the label distribution", which seems based on evaluation results (Table 6).
  7. Narrow scope and baselines. Only Qwen2.5-7B-Instruct is used. The 14B probe has one seed below the base model, and the Mistral transfer is called "unstable" with no numbers. All baselines are prompt variants on the frozen model, and the "style" rows are not the real Logic-LM or SymbCoT systems. There is no trained baseline, such as SFT on ProofWriter's gold proofs or iterative RPO (Pang et al., 2024). Building pairs from a correct and an incorrect answer to the same question is already standard, so the new part is only the ranking score that points 1–4 question.
  8. Presentation. Almost every section repeats the same caveats ("one deterministic output, no reward scoring, reranking, majority voting or oracle selector"), and it relies on many coined labels ("selection-policy alignment gap", "parser-safe same-scaffold settings", "RSPT-Scaffold@1", "low-/high-strength"). Figure 2's labels are too small to read in print and include the typo "parsrd answer". The text does not seem carefully edited.
Questions for authors
  1. How does Reward-rerank@8 do using only the recoverability terms, without the gold-answer gate?
  2. What is the average margin for the base model's traces, and for an empty or unrelated trace?
  3. What share of the rejected training traces fail the parser or the format check? What are the parse-failure rates for every frozen and adapted row?
  4. Can you run random, answer-only and slot-only pairs with three seeds on both datasets, keep the format and copy terms in the slot-only variant, and test each against full RSPT?
  5. Why do Table 2's Proof-first numbers differ from Table 9?
  6. How were the learning rates and the global scaffold chosen, and on which data? Which ProofWriter subset and depth do you use?
  7. Does RSPT improve proof correctness against ProofWriter's gold proofs?
  8. How does RSPT compare with SFT on gold proofs, with iterative RPO, and on a stronger backbone? What are the Mistral numbers?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Utyi (1/3)

We thank the reviewer for the detailed and constructive assessment, and for recognizing RSPT's inexpensive inference and the controlled same-scaffold evaluation. We agree that the original experiments did not isolate recoverability-guided preference construction from conventional answer-filtered training, and that gold-gated reranking cannot validate the semantic score. We added matched three-seed controls, ungated reranking, trace-dependence and margin analyses, parser/copy audits, and a second-backbone replication (mainly Appendix B.12).

W1 / Q4. Does recoverability-guided ranking improve over ordinary preference construction?

We agree that the 9/9 frozen-versus-adapted comparison does not isolate the score. We agree, too, that the original FOLIO tests against Base are not significant (p = 0.20–0.34), and that two FOLIO gains are within noise. RQ1 now describes the 9/9 result as positive mean gains and states that the two smallest gains are within or near seed variation.

We therefore ran matched three-seed experiments (seeds 7/42/123). All arms share the training questions, sampled candidate pools, correctness eligibility, rejected traces, optimizer configuration, and evaluation protocol; only the chosen-trace rule changes. NO_MARGIN is the slot-only variant you requested: it keeps SgoldS_{\mathrm{gold}} and the original format/copy treatment and removes only the corrupted-slot margin.

Qwen2.5-7B, paired gain (pp)FOLIOProofWriter
FULL − AO+4.90, 95% [+1.50, +7.17]+2.11, 95% [+0.22, +4.18]
NO_MARGIN − AO+2.94, 98.33% [+0.13, +6.32]+1.61, 98.33% [+0.06, +2.70]
FULL − NO_MARGIN+1.96, 98.33% [−0.54, +4.67]+0.50, 98.33% [−0.56, +1.84]

All FULL−AO seed differences are positive on both datasets. Most of the gain is retained by SgoldS_{\mathrm{gold}} alone. FULL−NO_MARGIN has one negative seed per dataset, so an independent margin contribution is not established.

We also ran standard two-sided construction (chosen and rejected both by answer order, or both random):

Qwen accuracy (mean ± SD)FOLIOProofWriter
FULL67.16 ± 0.4966.11 ± 1.51
Standard ORDER_BOTH65.69 ± 0.4965.17 ± 1.33
Standard RANDOM_BOTH64.22 ± 0.4964.94 ± 1.11

FULL is higher in every seed on both datasets, but the 97.5% adjusted intervals include zero, so we claim a directional advantage only. We keep the counterexamples you cite: in the original single-configuration runs, Random (68.83%) and Answer-only Type-A (68.17%) exceed full RSPT (66.11%) on ProofWriter, and simple controls also lead on ProntoQA. Those controls were not matched, and under matched three-seed training the random construction is below FULL in every seed. The matched FULL mean (66.11%) coincides numerically with the original mean but comes from distinct runs (Table 14).

Second backbone. With independently sampled pools and newly trained adapters on Llama-3.1-8B-Instruct (no Llama-specific tuning), FULL−AO is +4.08 pp on FOLIO (95% [+0.13, +7.78]) and +2.67 pp on ProofWriter (95% [+0.28, +5.08]).

Intervals use 20,000 context-cluster bootstrap resamples (73 FOLIO / 345 ProofWriter groups) and are conditional on the fitted adapters (Appendix B.12.2–B.12.4, B.12.9; Tables 14, 15, 22).

W2 / Q1. Reward-rerank@8 without the gold-answer gate

You are correct: Reward-rerank@8 maximizes Eq. (4) including the gold-answer indicator, so matching Oracle@8 says little about recoverability. We removed only the indicator, keeping the semantic, format, and copy terms and their coefficients (Eq. 17, Appendix A.8), on the same eight-candidate pools:

SelectorFOLIOProofWriter
Majority@863.7360.00
Ungated score@864.2255.00
Gold-gated Reward-rerank@873.5385.50
Oracle@879.4185.50

Ungated reranking is one question above Majority@8 on FOLIO and 5.00 pp below it on ProofWriter (Table 16). We have removed the claim that reranking validates the score. RSPT uses recoverability only to rank already answer-correct training candidates.

AuthorsOpenReview ↗

Response to Reviewer Utyi (2/3)

W3 / Q2. Margins, trace dependence, and lexical overlap

Absolute margins (Appendix B.12.11, Table 23). We scored frozen-model traces and three controls with the frozen scorer, with Mslot=Sgold−ScorrM_{\mathrm{slot}} = S_{\mathrm{gold}} - S_{\mathrm{corr}}:

TraceFOLIO mean MFOLIO P(M>0)PW mean MPW P(M>0)
Real frozen trace−0.078919.61%−0.022023.50%
Empty−0.105214.22%−0.021920.83%
Unrelated−0.117010.29%−0.026719.17%
Length-matched unrelated−0.122011.76%−0.027818.00%

You are right that the absolute margin is negative in every condition. Only 19.61% of FOLIO and 23.50% of ProofWriter real traces have a positive margin. Paired real − control margin differences (95% context-cluster CIs) are:

  • FOLIO: +0.026 [0.014, 0.040] vs. empty, +0.038 [0.022, 0.056] vs. unrelated, +0.043 [0.025, 0.066] vs. length-matched.
  • ProofWriter: −0.0001 [−0.002, 0.002] vs. empty, +0.005 [0.002, 0.007] vs. unrelated, +0.006 [0.004, 0.008] vs. length-matched.

The margin is therefore trace-sensitive on FOLIO but effectively trace-independent relative to an empty trace on ProofWriter. This is consistent with the matched ablation: the margin's training contribution is not established.

SgoldS_{\mathrm{gold}} is what separates traces, and it does depend on the trace (Table 17, nats/token):

  • FOLIO: real exceeds empty by +0.248, unrelated by +0.465, and length-matched unrelated by +0.479 [+0.428, +0.532].
  • ProofWriter: +0.012, +0.135, and +0.139 [+0.129, +0.148].

The small ProofWriter gap over an empty trace is now stated as a limitation in RQ4.

Restating the problem. SgoldS_{\mathrm{gold}} correlates only moderately with input/trace n-gram overlap (Spearman ≈ 0.19–0.28; Table 18). On questions with at least two eligible traces, SgoldS_{\mathrm{gold}} and 5-gram-overlap selection agree on 35.46% (FOLIO) and 40.84% (ProofWriter) of choices, vs. 25.91% and 26.47% expected by chance. In matched training, NO_MARGIN exceeds the OVERLAP selector in every seed (+2.45 and +1.17 pp), but the 97.5% intervals [−0.66, +5.42] and [−0.76, +3.19] include zero. FULL adapters also trigger the copy diagnostic more often than AO (below), so we do not claim that restating is excluded.

Slot quality. A metadata-substitution probe (4,808 instances, 14,424 gold-label replacements) left all scorer inputs unchanged. We have not audited extraction or corruption accuracy, so we limit the claim to scorer-relative recoverability under the fixed slot interface.

W4 / Q3. Rejected traces and parse failures

You correctly identify the gating structure. For an incorrect trace, R=λfFfmt−λcCcopyR = \lambda_f F_{\mathrm{fmt}} - \lambda_c C_{\mathrm{copy}}, so rejected selection depends only on format and copy; Sec. 3.3 and Appendix A.3 now state this, with the tie-break rule.

Training-pool audit (Table 20)FOLIOProofWriter
Selected rejected: parser failure2/339 (0.59%)1/1,777 (0.06%)
Selected rejected: copy trigger20/339 (5.90%)145/1,777 (8.16%)
Pair-eligible incorrect pool: copy trigger39/1,310 (2.98%)244/7,107 (3.43%)

Parse failures are rare among rejected traces, but copy-penalized traces are over-represented.

Two further points:

  • Matched outputs (Table 19). The new Qwen FULL and AO adapters have 0/612 (FOLIO) and 0/1,800 (ProofWriter) parse failures. Because the matched experiments also fix every rejected trace, the FULL−AO gains cannot come from rejected-trace format or parseability. FULL does produce more copy-triggered outputs than AO (20 vs. 10/612; 106 vs. 55/1,800), so we do not claim that RSPT reduces copying.
  • Original Table 2 rows. The original audit reports the ≤5% inclusion rule and the 11.67% frozen ProofWriter SymbCoT-style rate. Exact per-row rates for the other original scaffolds were not retained, and we do not substitute the matched-adapter rates for them.
AuthorsOpenReview ↗

Response to Reviewer Utyi (3/3)

W5 / Q7. Proof correctness against ProofWriter gold proofs

We agree that accuracy and scorer-based recoverability are not proof correctness, and that Table 3 uses a training-related scorer; it is now labeled descriptive. Appendix B.12.8 removes the subset confound by comparing FULL and AO on jointly correct examples, where FULL has higher SgoldS_{\mathrm{gold}} in every seed. This still uses the same scorer. We have not evaluated generated proofs against ProofWriter's gold proofs, so we make no proof-correctness claim. Accordingly, the "faithful proof" wording is replaced by scoped recoverability claims in the abstract, method, and conclusion.

W6 / Q5–Q6. Protocols, reported values, and data

Table 2 vs. Table 9. The same three adapter checkpoints were evaluated under two prompts, both with Qwen2.5-7B-Instruct, the same parser, greedy decoding, and a 768-token cap:

  • Table 2 uses a two-line Reason / Final answer prompt.
  • Table 9 (Appendix B.7) uses a six-line Entities / Predicates / Rules / Target / Derivation / Final answer prompt.
Table 2: Frozen → RSPTTable 9: Frozen → RSPT
FOLIO61.76 → 66.67 ± 0.4962.75 → 66.34 ± 0.28
ProofWriter56.83 → 67.28 ± 0.5958.00 → 66.11 ± 0.79

The McNemar tests in Appendix B.6 apply only to the six-line protocol. Sec. 4.1 and Appendix B.7 now say so.

66.67 / 66.11 / 66.18. The original FOLIO controls were single runs with seed 123, so they are compared with the seed-123 full run (66.67%, which is also the highest of 66.67/66.18/66.18). ProofWriter reports the three-seed mean. The 66.18% in the slot-sensitive analysis comes from a separate single run. These values are now labeled in the text, in Figure 4, and in Appendix B.9; the matched controls use three-seed means throughout.

Configuration selection. You are right that this used evaluation results. The main ProofWriter setting (learning rate 1×10−51\times10^{-5}, SimPO γ=0.3\gamma = 0.3) replaced the high-strength recipe (5×10−55\times10^{-5}, γ=0.1\gamma = 0.1) after the latter reduced Unknown accuracy from 55.50% to 29.00%. The global scaffold was also chosen during development. There is no untouched test split, so all results, including the new ones, are now described as development-set evidence (Sec. 4.1, Appendix B.12.1).

Data.

  • ProofWriter: 1,777 pairs come from 3,000 questions in the official V2020.12.3 OWA depth-5 meta-train.jsonl. Evaluation uses a label-balanced 600-question subset of the separate meta-dev.jsonl (200 per label; QDep 164/131/90/78/64/73 for depths 0–5). The original sampling seed was not preserved, but all questions are matched to official IDs, and the ID list will be released.
  • FOLIO: pairs (339) come from the official 1,004-example training split, and evaluation uses the 204-example validation split.
  • Normalized context hashing finds no exact train/evaluation match.

W7 / Q8. Baselines, backbones, and Mistral

We agree that same-question correct–incorrect pairing is standard (e.g., Pang et al., 2024). The contribution is the chosen-trace ranking signal, and the matched AO, NO_MARGIN, OVERLAP, and two-sided controls now test exactly that, under one SimPO/LoRA framework. We have not run SFT on gold proofs or full iterative RPO; the original chosen-only SFT uses model-generated traces, so it is not equivalent.

  • Second backbone: Llama-3.1-8B FULL−AO is +4.08 / +2.67 pp (see 1/3).
  • Qwen2.5-14B (FOLIO): 75.00, 71.57, and 68.14 (mean 71.57 ± 2.80) vs. frozen 69.12. One seed is below Base, so we claim no consistent scaling gain.
  • Mistral-7B-v0.3 (FOLIO, single seed, 192-token budget): frozen 46.57%; RSPT 47.06%, 46.57%, and 44.61% at learning rates 5×10−55\times10^{-5}, 1×10−51\times10^{-5}, and 5×10−65\times10^{-6} (Appendix B.12.10). This is not a comparable replication.
  • "Style" rows are controlled prompt variants, not the full Logic-LM or SymbCoT systems; Sec. 4.1 states this.

W8. Presentation

We reduced the repeated inference-time caveats and removed the "selection-policy alignment" framing. Figure 2 is corrected: the "parsrd" typo, the Reward-rerank@8 label, the incorrect-set definition (which now includes unparseable outputs), and the reward components (which now include SgoldS_{\mathrm{gold}}). Its font size is unchanged, and we will enlarge it in the final version.

Summary

The revised evidence supports a narrower conclusion: trace-conditioned gold-slot recoverability improves chosen-trace selection among answer-correct candidates, with positive matched FULL−AO results on both datasets and two backbones. We do not claim superiority over all two-sided strategies, an independent margin benefit, label-free verification, or improved gold-proof validity. Thank you for a critique that substantially sharpened the paper.

Reviewer_UtyiOpenReview ↗

Official comment by Reviewer Utyi

Thank you for the detailed rebuttal and for conducting the additional experiments. I appreciate the effort the authors have put into addressing my concerns.

The new results provide useful evidence that slot-based trace selection improves over answer-only selection. However, the independent contribution of the recoverability margin remains unclear: most of the improvement is retained by NO_MARGIN, and the FULL versus NO_MARGIN differences are not statistically significant on either dataset. The negligible margin difference between real and empty traces on ProofWriter further calls into question the role of the contrastive component.

The lack of direct proof-correctness evaluation and an untouched test set also remains a concern. While the additional Llama-3.1-8B results provide useful cross-backbone evidence, the broader evaluation remains limited. The Qwen2.5-14B results do not show a consistent gain, and the Mistral experiment is only a single-seed result. Moreover, the comparisons with existing methods such as Logic-LM and SymbCoT use controlled prompt variants rather than full-system implementations. Comparisons against gold-proof SFT and full iterative RPO would also help establish the method's value relative to alternative training approaches.

The presentation could also be improved; for example, Figure 2 is too small, making its details difficult to read.

Overall, despite the authors' efforts to address my concerns and narrow their claims, I am not fully convinced that the paper's central contribution is sufficiently established or compelling for acceptance in its current form. I will therefore retain my original score.

Reviewer_MkiaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RSPT constructs preference pairs for proof-first logical reasoning using a frozen model's likelihood of recovering problem slots from generated traces. Correct traces are ranked using gold-versus-corrupted slot scores, format checks, and copying penalties, then used for SimPO/LoRA training. Evaluation on FOLIO and ProofWriter reports improvements across nine controlled prompt/dataset settings under greedy single-output inference, alongside reward-reranking and mechanism controls.

Critical strengths
  1. The same-scaffold comparisons hold the backbone, prompt, parser, and decoding protocol fixed. Improvements across several scaffolds and three adapter seeds support the practical usefulness of adaptation without adding test-time selection. The paper clearly distinguishes this deployment setting from multi-sample analysis.

  2. The controls address plausible alternative explanations, including answer filtering and exposure to selected correct traces. The input-defined FOLIO stress subset gives additional evidence for the proposed signal. Reporting stronger simple controls on ProofWriter and the Unknown-label trade-off usefully limits the conclusions.

Critical weaknesses
  1. The reward-validation experiment does not isolate semantic recoverability. Reward-rerank@8 maximizes the reward in Eq. (4), which explicitly uses the gold-answer correctness indicator. Its advantage over majority voting and proximity to Oracle@8 therefore cannot establish that the reverse-semantic score identifies better traces. This affects the analysis, rather than demonstrating label leakage in single-output inference. The separate recoverability audit also uses the training scorer and averages over each method's own correctly answered examples, mixing trace quality with differences in example composition.

  2. Evidence for an advantage over simpler preference construction remains limited. On low-strength ProofWriter, random Type-A reaches 68.83%, above full RSPT at 66.11%. FOLIO favors RSPT, but the controls lack seed variability and paired comparisons against RSPT; the reported FOLIO tests against Base are not significant. Moreover, Table 2 and Appendix B.7 report different proof-first/Base results without explaining the configuration difference. For ProofWriter, these frozen baselines are 56.83% and 58.00%, respectively. This prevents transferring the appendix's significance evidence directly to the headline comparisons.

Questions for authors
  1. Does Reward-rerank@8 actually use the full gold-gated reward? On the existing candidate pools, what happens when selection uses only the semantic-score terms, with the same format and copying treatment? For the recoverability audit, can methods be compared on a shared set of examples?

  2. For the existing FOLIO runs, which seeds and training questions were shared by RSPT, random Type-A, and answer-only Type-A? What is the paired uncertainty of their accuracy differences on the saved predictions?

  3. Which prompt/checkpoint configurations explain the differences between Table 2 and Appendix B.7? Please also identify the source train/evaluation partitions, the ProofWriter subset-selection procedure, and how preference-training examples and evaluation contexts were kept separate.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Mkia (1/2)

We thank the reviewer for the careful assessment and precise questions. We added ungated reranking, a shared-example scorer audit, matched three-seed controls with paired uncertainty, and explicit protocol and data-provenance disclosures (mainly Appendix B.12).

Q1 / W1. Gold-gated reranking, ungated selection, and shared-example audit

Does Reward-rerank@8 use the full gold-gated reward? Yes. It maximizes the full reward in Eq. (4), including the gold-answer correctness indicator. We agree that its advantage over Majority@8 and proximity to Oracle@8 cannot establish that semantic recoverability independently identifies correct answers.

We now state this in Sec. 4.3, Appendix A.8, and the Figure 2/3 captions.

What happens without the gate, keeping the semantic, format, and copying terms unchanged?

We remove only the correctness indicator, retaining the original coefficients:

Rfree=1+λmMslot+λgSgold+λfFfmt−λcCcopyR_{\mathrm{free}} = 1+\lambda_m M_{\mathrm{slot}}+\lambda_g S_{\mathrm{gold}}+\lambda_f F_{\mathrm{fmt}}-\lambda_c C_{\mathrm{copy}}

This is Eq. (17) in Appendix A.8, evaluated on the same saved eight-candidate pools.

SelectorFOLIOProofWriter
Original@161.2754.83
Majority@863.7360.00
Ungated score@864.2255.00
Gold-gated Reward-rerank@873.5385.50
Oracle@879.4185.50

Ungated selection exceeds Majority@8 by one example (+0.49 pp) on FOLIO and trails it by 5.00 pp on ProofWriter (Appendix B.12.5, Table 16).

We therefore do not claim label-free correctness verification. RSPT uses semantic scoring to rank candidates already identified as answer-correct during training. Single-output adapted inference requires neither gold labels nor reward scoring.

Can recoverability be compared on a shared set of examples?

Yes. Table 3 compares method-specific correct subsets, which confounds trace scores with example composition. We now compare FULL and matched answer-order (AO) adapters on exactly the examples both answer correctly, separately for each seed (Appendix B.12.8, Table 21).

For seeds 7/42/123, FULL−AO gold-slot score differences are +0.125/+0.096/+0.096 nats/token on FOLIO (116/109/116 jointly correct examples) and +0.052/+0.035/+0.057 on ProofWriter (336/344/334).

The slot-margin differences are also positive but small in every seed. The audit still uses the training scorer, so we interpret it descriptively rather than as independent proof-validity evidence.

W2. Advantage over simpler preference construction

We agree that the original controls alone did not isolate recoverability-guided chosen-trace selection. We therefore added new matched three-seed controls (7/42/123) that fix training questions, candidate pools, correctness eligibility, rejected traces, optimizer configuration, and evaluation protocol. Only the chosen-trace selection rule changes.

With Qwen2.5-7B-Instruct:

Paired comparison (pp)FOLIOProofWriter
FULL − AO+4.90, 95% CI [+1.50, +7.17]+2.11, 95% CI [+0.22, +4.18]
NO_MARGIN − AO+2.94, 98.33% CI [+0.13, +6.32]+1.61, 98.33% CI [+0.06, +2.70]
FULL − NO_MARGIN+1.96, 98.33% CI [−0.54, +4.67]+0.50, 98.33% CI [−0.56, +1.84]

FULL outperforms AO in every seed on both datasets. Independently sampled candidate pools and newly trained adapters with Llama-3.1-8B-Instruct also yield positive FULL−AO gains: +4.08 pp on FOLIO (95% CI [+0.13, +7.78]) and +2.67 pp on ProofWriter (95% CI [+0.28, +5.08]).

What drives the gains? Removing the corrupted-slot margin retains most of the improvement over AO, while the incremental FULL−NO_MARGIN effect is not statistically established. The evidence therefore supports trace-conditioned gold-slot likelihood as the main ranking signal, not an established contribution of the margin.

Regarding the ProofWriter counterexample raised in W2: the original Random Type-A result (68.83%) indeed exceeds the original FULL mean (66.11%), and we retain this finding.

However, in the new three-seed comparisons, FULL exceeds standard random two-sided preference construction in every seed on both datasets:

  • FOLIO: 67.16% versus 64.22% (+2.94 pp).
  • ProofWriter: 66.11% versus 64.94% (+1.17 pp). (This matched FULL mean coincides numerically with the original six-line mean but comes from distinct runs; Table 14.)

Standard two-sided controls change both the chosen and rejected traces, unlike matched AO. Their multiplicity-adjusted 97.5% confidence intervals include zero; hence we report positive mean differences, not statistically established superiority over conventional two-sided construction.

We also acknowledge that the original FOLIO McNemar tests against the frozen Base were not significant. The revised paper describes the original 9/9 result as positive mean gains, not nine statistically significant improvements.

Full results are in Appendix B.12.2–B.12.4, Tables 14–15.

AuthorsOpenReview ↗

Response to Reviewer Mkia (2/2)

Q2. Shared seeds and training questions in the original FOLIO controls

The original FOLIO FULL RSPT, Random Type-A, and Answer-only Type-A experiments all used seed 123 and were evaluated on the same 204 validation questions.

Their preference-training data originated from the official FOLIO training split. However, those runs did not fully control candidate pools, selected training-question subsets, or rejected-trace selection across methods.

Consequently, sharing seed 123 and the evaluation questions does not make these runs a fully matched experiment. The single-run FULL advantage over Random Type-A (66.67% vs. 63.24%, seven questions) and Answer-only Type-A (61.27%, eleven questions) therefore cannot be attributed uniquely to the ranking rule.

We retain these single-run comparisons as descriptive evidence rather than treating them as isolated causal tests.

To directly address the requested paired uncertainty, we conducted new three-seed experiments (7/42/123), fixing training questions, candidate pools, correctness eligibility, rejected traces, optimizer recipe, and evaluation protocol.

FULL−AO achieves +4.90 pp on FOLIO (95% CI [+1.50, +7.17]) and +2.11 pp on ProofWriter (95% CI [+0.22, +4.18]).

The confidence intervals use 20,000 context-cluster bootstrap resamples over 73 FOLIO and 345 ProofWriter context groups. The primary comparisons use 95% intervals; the 98.33% and 97.5% intervals apply Bonferroni adjustments for their respective comparison families.

All intervals are conditional on the fitted adapters and fixed development subsets.

Q3. Table 2 versus Appendix B.7, checkpoint configurations, and data provenance

Checkpoint and prompting protocols.

We confirm that the same three trained adapter checkpoints were evaluated under both prompting protocols.

Main Table 2 uses a compact two-line Reason / Final answer prompt. Appendix B.7 (Table 9) and the original mechanism analysis use a six-line slot-structured prompt:

Entities / Predicates / Rules / Target / Derivation / Final answer.

Both protocols use Qwen2.5-7B-Instruct, the same final-answer parser, greedy one-output decoding, and max_new_tokens=768.

Their different frozen baselines arise from the different evaluation prompts:

DatasetTable 2 (two-line)Appendix B.7 (six-line)
FOLIO frozen61.76%62.75%
ProofWriter frozen56.83%58.00%
FOLIO adapted mean66.67%66.34%
ProofWriter adapted mean67.28%66.11%

Although the adapter checkpoints are the same, the predictions differ under the two prompts. Thus, the McNemar tests in Appendix B.6 (Table 8) apply only to the six-line protocol and cannot be transferred to Table 2.

We have clarified the protocol distinction in Sec. 4.1 and Appendix B.7.

ProofWriter partitions and subset selection.

The 1,777 preference pairs were constructed from 3,000 source training questions in the official ProofWriter V2020.12.3 OWA/depth-5 meta-train.jsonl.

Evaluation uses a fixed, label-balanced subset of 600 questions from the separate official meta-dev.jsonl, not meta-test.jsonl.

The subset contains 200 questions per label, with QDep counts of 164/131/90/78/64/73 for depths 0–5. The original sampling seed and sampler were not preserved, but the evaluated questions were matched to official question/context identifiers. The ordered identifier list is designated for release with the code.

FOLIO partitions.

FOLIO preference-training examples originate from the official training split, which contains 1,004 examples. After candidate generation and correctness-based eligibility filtering, 339 Type-A preference pairs were constructed. Evaluation uses the separate official validation split containing 204 examples. Native story IDs were not preserved throughout the processed preference pool, so we additionally checked separation using normalized context hashes.

Train/evaluation separation and development-set limitations.

A conservative normalized context-hash comparison found zero exact training/evaluation context matches on both benchmarks.

Both evaluation subsets were also used for scaffold or training-configuration selection. We therefore explicitly describe all reported results, including the new matched comparisons, as development-set evidence rather than independent held-out test confirmation.

The relevant disclosures appear in Sec. 4.1 and Appendix B.12.1.

Closing

The new experiments support a narrower and more precise conclusion: among answer-correct candidate traces, recoverability-guided selection improves training-time preference construction relative to a matched answer-order baseline.

We do not claim that gold-gated reranking validates label-free correctness selection, that the contrastive margin has a separately established benefit, or that the recoverability score independently certifies proof correctness.

Comments on the paper

1 comment

Dear Reviewer Utyi and Mkia,

Please take a moment to review and respond to the authors' rebuttal to facilitate further discussion, as the author-reviewer discussion period ends on November 18 at 11:59 PM AoE.

Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.