- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Reasonable-rlvr trains a model to answer questions with cited proof steps. The input includes a question and graph facts that contain the answer. The verifier checks claims against graph closure and uses a relation contract to check whether the supported chain reaches the answer. Only the verifier sees this contract. Training combines answer correctness with process rewards from these checks. The reported correct & valid rates are higher than with outcome rewards on 4 main benchmarks and smaller graphs.
Critical strengths
(1) The checker separates supported facts from a chain that establishes the answer. Table 12 explains the disagreements with the independent checker. These come from answer parsing, repeated ids and related specification choices. No closure disagreement was found in that audit. This helps show what is still unclear.
(2) It is a good experimental design choice to use a common verifier for the reward comparisons. The outcome+format control helps check whether gains come mainly from learning the output format. The 8 sample eval also gives useful support beyond the greedy comparison.
(3) On Webqsp 2 seeds reach the same reward but only 1 collapses. Some runs also recover after a period of producing only one claim. The ordering tests alone do not ensure stable training: Cou passes the tests yet learns less reliably than Cdu. The adaptive Webqsp runs reduce collapse.
Critical weaknesses
(1) The main benchmarks have no class declarations, transitivity or property chain rules. The richer ontology layer therefore has limited direct support in these results.
(2) The contract comes from a reference relation sequence. It is unclear how this covers CWQ constraints outside the chosen chain. The independent checker checks the same specification rather than whether the contract covers the full question. This leaves the broader reasoning claim less clear. It does not show that an invalid proof was accepted.
(3) The coverage result may depend on how a reversal is written. On Dynamic-kgqa, an explicit hidden reverse triple still hurts soundness under abstention. A forward fact used backwards keeps its support. In masked edge experiment, masked claims leave the soundness denominator and can count towards chain/progress. The protocols are described. It would be good to use trace counts to explain how much this difference affects recovery.
Questions for authors
please see weakness part.
4 comments
Thank you again for your careful review. We hope our reply and the revision address your points. In brief:
- (1) Ontology layer. Test sets on which a valid proof must use a declared inverse or transitive rule, type conditions from each source's own axioms, and 4,513 verified proofs that depend on a transitivity or chain rule (App. M, Tables 55 and 56).
- (2) CWQ constraints. A direct check of the compared date or count on 239 constraint questions, where the proof-trained policies pass on .46 to .69 against .00 to .18 for outcome (Table 62).
- (3) Reversals and masked claims. Trace counts show that the recovery comes from true facts the training verifier cannot see, and abstaining stays at or above strict when both directions of a masked relation are hidden (Tables 59 to 63).
If these results resolve your concerns, we would be grateful if you would consider updating your assessment. We are glad to answer any further question before the discussion closes.
Thanks for the detailed response! which address all my concerns and I have raised my score to accept accordingly.
We are glad our responses addressed your concerns. Thank you for the thoughtful review and for your endorsement of our work.
We thank the reviewer. All results are new and in the revised PDF.
(1) The ontology layer in the main benchmarks.
How we read the concern. The four benchmarks declare no classes, transitivity or property chains, so the main results give little direct evidence for the richer rule layer.
(a) Rules a proof must use. We built test sets on which a valid proof needs a declared rule (Table 55, three seeds). With the inverse fact removed from Dynamic-KGQA blocks, the proof-trained policies prove .51 to .58 of the 2,958 questions by citing the forward fact, which the checker accepts through the declared inverse (.006 without it). On 235 MINTQA questions answerable only through a transitive run of two located-in or part-of facts, CWQ-trained CDU policies, never trained on such questions, prove .39 to .49 at 1.5B to 7B and cite both facts in .63 to .82 of replies against .38 to .44 for outcome (0 with transitivity off). With a warm-up that writes the rule as a step, the policies write the inverse rule explicitly on .94 to .96 of the inverse questions and keep .47 to .51 with inverse reading off (App. M).
(b) Classes and types. With each source's own classes and domain and range axioms added to the checker, the type condition rejects 107 and 537 proofs whose answer is the wrong kind of entity and keeps every correct-and-valid verdict (App. M). Property chains are declared on UMLS, CoDEx-S and the synthetic graphs, where the checker verifies 4,513 proofs that depend on a transitivity or chain rule (Table 56).
(2) CWQ constraints outside the chain, and the independent checker.
How we read the concern. The contract follows the reference relation chain, so a comparative or superlative constraint may lie outside it, and agreement with the independent checker is agreement on the same specification.
(a) What is enforced. The correct-and-valid rate requires both a valid chain and a correct answer, so the constraint is enforced through the answer. On 1,000 mutated reference proofs per benchmark (Table 58), both checkers reject every swapped object, dropped step, fabricated identifier, restated premise and wrong symmetry rule on CWQ, Dynamic-KGQA and WebQSP, and the answer key rejects every proof that moves the answer to another entity on the chain. On the constraint questions, 182 of 192 CDU and 190 of 199 gated valid proofs that end at a wrong answer name an entity that loses the comparison or is not a compared candidate, and the answer key rejects every one (App. M).
(b) Checking the constraint itself. For 239 constraint questions we linked the compared date or count from the 2015 Freebase dump, added the 388 facts to the evidence block, and required the proof to cite the answer's attribute fact and the comparison to hold (Table 62, three seeds pooled, n = 717). Without any training on such facts, the proof-trained policies pass on .46 to .69 of questions at 1.5B, 3B and 7B, against .00 to .18 for outcome (p below 1e-40), and cite the answer's attribute fact in .62 to .89 of replies.
(c) Contracts from the question. Training with the question's endpoints and a hop bound instead of reference relations gives .729 (CDU) and .712 (gated) against .704 and .708 (App. H).
(3) Reversals, masked claims and the soundness denominator.
How we read the concern. Under abstention a masked claim leaves the soundness denominator in training, and a reversal can be written two ways. The reviewer asks for trace counts showing how much this contributes to the recovery. Evaluation always uses the full verifier, so a claim set aside in training counts only if it is true.
(a) Trace counts. We counted valid evaluation proofs that need a claim on a relation hidden in training (Table 59). At 0.5B, where the paper reports the recovery, abstaining training kept such proofs on the toy graph (10 of 18, 12 of 19 and 9 of 16 over three seeds) while strict training suppressed them (0 of 7, 3 of 12, 0 of 0), and every such claim is confirmed by the full verifier. The recovery is the policy continuing to write true facts the training verifier cannot see.
(b) Training credit. In the stored training rollouts of a three-seed 0.5B replay (8,000 per cell), a false masked claim earns establishment credit in at most 6 rollouts per cell and never earns progress credit, and in the table runs 12,684 of 12,688 masked claims are true (Table 61, App. M).
(c) Reversal form. On Dynamic-KGQA most replies write both forms, and the strict and abstaining policies distribute over them alike (Table 60). Hiding both directions of every masked relation, which removes the forward-fact route, keeps abstaining above strict at 0.5B (.279 against .197, p = .03, and .803 against .439, p = 2e-5) and at or above it at 1.5B (Table 63). At 3B over seven seeds, abstaining recovers the full-verifier rate (.819 against .805, n = 343). The recovery does not rest on how the reversal is written.