- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed a train-time plug-in method, termed Reverse-Semantic Preference Training (RSPT), for proof-first logical reasoning with proof-trace. It samples candidate traces, extracts deterministic semantic slots, constructs corrupted slot alternatives, and apples the frozen backbone to score the influence of the trace. Experimental results across FOLIO and ProofWriter demonstrates the effectiveness of the proposed method.
Critical strengths
This paper is well organized. The proposed reverse-semantic preference training method with trace selection, semantic slot extraction, and trace evaluation is interesting.
Critical weaknesses
-
The proposed RSPT scores whether the selected semantic slots can be recovered from a trace. However, a trace which preserve these slots may contain invalid or less information for reasoning. Improving the recoverability does not demonstrate that the generated proofs can be logically correct or the trace can be effective. Therefore, it is recommended to discuss the influence of the trace, the semantic slots, and the semantic recoverability independently in three steps.
-
The experiments were carried out on only one backbone (Qwen2.5-7B-Instruct) and two typical benchmarks. Ablation studies on the proposed RSPT with different LLMs, different semantic recovering strategies, and different trace selection strategies are recommended. More comparison studies on the state-of-the-art techniques with more benchmarks are also suggested.
Questions for authors
/
2 comments
Response to Reviewer SVsU (2/2)
W2. Additional backbones, recovery strategies, and selection baselines
Different LLM backbones. We independently sampled candidate pools and trained new adapters on Llama-3.1-8B-Instruct, without Llama-specific hyperparameter tuning. FULL exceeds matched answer-order selection by +4.08 pp on FOLIO (95% CI [+0.13, +7.78]) and +2.67 pp on ProofWriter (95% CI [+0.28, +5.08]), a second-model-family replication of the chosen-trace ranking effect (Appendix B.12.9). We also report a seed-sensitive Qwen2.5-14B diagnostic and an earlier single-seed Mistral probe that does not establish reliable transfer (Appendix B.10, B.12.10). We therefore do not claim universal backbone robustness.
Different scoring and trace-selection strategies. Under matched conditions, we compare the following chosen-side rules:
- FULL;
- NO_MARGIN, which uses gold-slot likelihood only;
- answer-order selection;
- 5-gram-overlap selection.
We also evaluate standard random and answer-order two-sided preference construction. FULL has higher accuracy than both two-sided controls in every Qwen seed on both datasets. The adjusted confidence intervals include zero, so we report directional gains rather than statistically established superiority (Appendix B.12.3–B.12.4). These comparisons separate recoverability scoring from candidate order and lexical overlap. They are not an exhaustive comparison of alternative slot-extraction designs.
Benchmarks and baselines. FOLIO and ProofWriter remain our two primary benchmarks. ProntoQA is a supplementary diagnostic, and there simpler preference controls outperform RSPT. Our prompting baselines (Logic-CoT-, SymbCoT-, and COP-style) are controlled variants under a shared parser and decoding protocol, not full reimplementations of those systems. Broader benchmark coverage, alternative slot representations, stronger full-system baselines, and verified proof correctness remain important extensions, and we state them as limitations.
Summary
The revised experiments separate trace dependence, semantic-slot scoring, and downstream preference-selection effectiveness. The best-supported conclusion is that trace-conditioned gold-slot likelihood provides a useful signal for selecting among answer-correct training traces, with matched positive results across two datasets and two model families. We have narrowed our claims accordingly. Recoverability is a training-time proxy, not a certificate of logical proof correctness, and most of the gain comes from gold-slot likelihood rather than the contrastive margin.
We thank the reviewer again for the constructive suggestions, which helped clarify both the empirical contribution and its boundaries.
Response to Reviewer SVsU (1/2)
We thank the reviewer for finding the paper well organized and the reverse-semantic preference method interesting. Both suggestions, separating the roles of the trace, the semantic slots, and recoverability, and broadening the evaluation, led to new controlled analyses (Appendix B.12, including the new margin analysis in B.12.11).
W1. Separating trace information, semantic slots, and recoverability-guided selection
We agree that preserving semantic slots does not guarantee logically correct or faithful proofs. Following your suggestion, we now address the three components separately.
(1) Does the score depend on the actual trace?
We compared real frozen-model traces with empty, unrelated, and length-matched unrelated traces, keeping the scorer and the slots fixed (Appendix B.12.6 and B.12.11).
For gold-slot likelihood Sgold, the score that drives selection, real traces score higher than all three controls on both datasets. Against the length-matched unrelated control, the gain is +0.479 nats/token on FOLIO (95% CI [+0.428, +0.532]) and +0.139 on ProofWriter ([+0.129, +0.148]). Against an empty trace, however, the ProofWriter gain is only +0.012.
For the contrastive margin Mslot=Sgold−Scorr (Table 23):
Real traces also raise the margin relative to the unrelated and length-matched controls on both datasets. However, the absolute mean margins are negative, and the ProofWriter real-versus-empty difference is effectively zero. The trace therefore clearly shapes Sgold. The margin, by contrast, is only weakly trace-sensitive and does not reliably separate gold from corrupted slots.
(2) What do the semantic slots contribute?
The slot extractor is deterministic and dataset-specific. It represents targets, entities, predicates, operators, and the available facts and rules (Appendix A.1).
These analyses separate the slot-based scorer from literal overlap, but they do not independently validate every extracted relation or guarantee proof validity.
(3) Does recoverability-guided ranking improve downstream reasoning?
We ran matched three-seed experiments that fix the training questions, sampled candidates, correctness eligibility, rejected traces, optimization settings, and evaluation protocol; only chosen-trace selection changes. With Qwen2.5-7B-Instruct, FULL RSPT exceeds matched answer-order selection by +4.90 pp on FOLIO (95% CI [+1.50, +7.17]) and +2.11 pp on ProofWriter (95% CI [+0.22, +4.18]), in every seed.
These findings support recoverability-guided training-trace selection, mainly through gold-slot likelihood. They do not establish that the resulting proof steps are formally correct.