- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VERST, a framework to obtain process rewards from a formal solver. The novelty lies in the simpler structure representation that can be formalized more easily and in the active path tracking.
Critical strengths
The presentation is okay, and the method seems sound. The performance outperforms the baseline in the experiments.
Critical weaknesses
Writing needs a lot of improvement, and some design choices are not specified or motivated. For example, the paper mentions in the introduction that it designs a simpler structure that can be formalized more easily, but it has not mentioned how and why it is easier to be translated. Besides, active facts and inferences seem to be another technical contribution, but the paper does not describe how it is implemented. Furthermore, it mentions “annotator is asked to stick to” in line 210, but I have no idea why it needs this annotator and how it works. Also, the consistency check in sec 5.4 does not seem to be mentioned in the methodology as well.
Questions for authors
- Do you have a human agreement rate with the LLM-judge in sec 5.3?
- I don’t understand Section 5.4. Are you checking the consistency between the original NL and the VERS structure? And I cannot follow the experiment you describe between lines 425 and 427. I think you need to explain why this is examining the consistency.
2 comments
Reply to Question 1
We sampled 20 of the 100 problems in Section 5.3, collected the responses of Base, Outcome RL and VERST on them (60 responses in total), and manually scored them with the same rubric. The table reports the absolute difference between the human score and the LLM score.
| Dimension | Mean | Std. | |||
|---|---|---|---|---|---|
| Validity | 82% | 95% | 98% | 0.25 | 0.60 |
| Faithfulness | 75% | 90% | 95% | 0.47 | 0.75 |
| Completeness | 78% | 93% | 97% | 0.32 | 0.62 |
| Conciseness | 67% | 85% | 97% | 0.55 | 0.80 |
| Average score | 25% | 98% | 100% | 0.37 | 0.33 |
On the average process score, 98% of the responses have and the mean difference is 0.37, which indicates that the LLM judge agrees closely with human scoring. The human scores of the three models also agree with the ranking of the LLM judge in Table 2 in paper:
| Model | Validity | Faithfulness | Completeness | Conciseness | Avg. |
|---|---|---|---|---|---|
| Base | 4.35 | 3.95 | 4.55 | 2.95 | 3.95 |
| Outcome RL | 4.15 | 3.70 | 4.45 | 2.50 | 3.70 |
| VERST | 4.55 | 4.20 | 4.80 | 2.95 | 4.12 |
We have added this agreement analysis to Appendix E.3.
Reply to Question 2
We apologize for the confusion. The consistency check is a component of the VERS Reward, and it checks the consistency between the VERS structure and the original natural-language reasoning, including whether each inference corresponds to a derivation the natural language actually uses, whether each fact is equivalent to the information used in the natural language, and whether the granularity of the structure matches that of the reasoning.
The motivation for the consistency check is to avoid reward hacking. Since the reward is computed on the VERS that the model itself generates, the model can learn during RL to generate a VERS that is inconsistent with its natural-language reasoning but still passes formal verification, which earns a higher reward without improving the reasoning. The VERS then no longer reflects the natural-language reasoning, the formal verification result becomes unreliable, and the quality of the natural-language reasoning does not really improve.
Section 5.4 is the ablation of this component, and lines 425–427 in original paper describe the design of the experiment. It records, at each epoch of RL, the training reward together with a reasoning quality that is assessed independently on the natural-language reasoning alone (Figure 6). Without the consistency check, the reasoning quality does not rise together with the reward. With the check, the two rise together, which supports our claim that the consistency check prevents reward hacking.
We sincerely thank you for the careful reading and the constructive comments. We accept your criticism on the presentation, and we apologize for not explaining the choice and motivation of each design clearly. Below we explain these designs one by one and report a human agreement check for the LLM judge. We have revised the paper according to your suggestions.
Reply to Weakness 1
Natural-language reasoning is free-form: a single step mixes several facts with the inferences among them and contains vague references and anaphora, so the translator does not translate individual sentences. It must first recover the structure of the reasoning and extract the relevant statements. VERS represents this structure with atomic facts and explicit inferences, so translating a VERS reduces to translating individual atomic statements, at which LLM-based translators are already reliable. We have revised the introduction to explain this motivation.
In Section 3.1.3 we introduced two notions, active facts and inferences, and consistency, which correspond to the two check components of the VERS Reward, namely the active inference check and the consistency check. They are also what Section 5.4 analyzes. Below we describe how the two components are implemented.
Both checks are critical for preventing reward hacking during RL. We have revised Section 3.1.3 to define the two components explicitly and added the ablation of the active inference check to Section 5.4.
Writing a correct VERS for a natural-language reasoning step is non-trivial, and we therefore use an LLM with strong language understanding to annotate the VERS structures, so that the resulting annotations are of sufficient quality to serve as training data for reasoning structure learning. The procedure is as follows: we sample step-by-step chain-of-thoughts with step markers from the base model, pass the whole chain to the annotator, the annotator returns a VERS that follows the definition for each step, and the structures are then merged after the corresponding steps to obtain the reasoning traces with VERS. We have revised Section 3.2.1 to describe this procedure.
We thank you again for pointing out these issues in the paper. If anything remains unclear, please let us know, and we would be happy to explain it further and to improve the paper.