- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds process rewards by checking intermediate artifacts, including citations, code execution records, image crops, and numerical equalities. A shared interface returns bounded credits assigned to interaction steps. These credits are combined with final-answer rewards for GRPO or PPO. Training starts with outcome rewards and then uses a random mixing weight shared by each prompt group. Experiments across four domains with Qwen3.5-4B and 9B report higher final accuracy than outcome-only GRPO. The aim is to improve intermediate reasoning and make agent behavior more reliable.
Critical strengths
-
The interface is simple and practical. It separates verification rules from reward composition and policy optimization. The same HotpotQA verifier improves both GRPO and PPO by 2.45 and 2.50 percentage points, respectively (Table 3). This gives concrete support for reuse across optimizers.
-
The evaluation covers four domains and two model sizes. Table 1 reports gains over outcome-only GRPO in all eight settings. For the 4B model, the method also exceeds the LLM-judge baseline on three of four tasks. These results provide useful evidence that the local checks can supply effective process supervision.
Critical weaknesses
-
Using process supervision to assign RL rewards is not new. For example, VPR uses symbolic or algorithmic verifiers to produce turn-level RL rewards, and VPRMs combine rule-based process rewards with outcome rewards. Tables 1–3 show that the complete system is effective, but do not include direct comparisons with these approaches. The LLM-judge baseline is useful, but does not establish an advantage over existing verifiable process-reward designs. The contribution beyond applying process supervision through a shared interface therefore remains unclear.
-
The evidence does not directly support the claims about long-horizon stability and self-reflection. Appendix C limits the interactive tasks to 3–5 turns. The main results measure final accuracy, and Figure 4 shows training accuracy. Neither directly measures reasoning consistency or correction behavior. Section 3.4 defines a joint outcome-and-consistency metric, but no results for it are reported. The reported gains support improved task performance in these settings, while the broader claims remain untested.
Questions for authors
-
What is the main technical advance over existing verifiable process-reward methods such as VPR and VPRMs? Could the authors provide a comparison in a compatible setting, using the same backbone, training budget, and available verification information, or explain why these methods are not directly comparable? This would help determine whether the gains come from the proposed design or from adding process supervision itself.
-
On the existing evaluation trajectories, how does the joint outcome-and-consistency metric in Equation (9) compare with outcome-only GRPO? Please use the same checks and evidence requirements for both policies. This would directly test whether higher final accuracy also comes with more consistent intermediate work.