acceptodds
Under review as a conference paper at ICLR 2027

Plug-and-Play Verifier for Decoupling Process Reward in Stabilizing Long-Horizon Agentic Reasoning

Abstract

Stabilizing long-horizon reasoning requires intelligent agents to sustain useful progress across successive reasoning steps and tool interactions. Such agents are typically trained with reinforcement learning (RL), where actor-critic RL uses a learnable critic network to estimate step advantages, while critic-free methods such as GRPO eliminate critic models and rely mostly on the group variants of sparse outcome-only rewards. Both reward types leave partially useful failed trajectories either implicit or indistinguishable from entirely unproductive ones, and as reasoning horizons grow, it becomes difficult to maintain consistent reasoning traces. In this work, we propose a plug-and-play verifier framework for stabilizing long-horizon reasoning through explicit verification of intermediate reasoning artifacts. Specifically, the framework decouples process verification through a unified interface that extracts intermediate reasoning artifacts from agent running trajectories, evaluates them using domain-specific verifiers, and aggregates the verification grades as bounded process reward. Then, we combine our dense verified process rewards and the sparse final outcome through a two-stage post-training schedule. We showed that with plug-and-play verifiers, we further stabilize the reasoning steps as last-mile self-reflection and self-consistency. Typical evaluation on HotpotQA using Qwen3.5-4B achieves a superior accuracy (60.18%), outperforming the standard GRPO baseline (57.73%). Our results suggest that plug-and-play verifiers provide a modular approach to learn more stable and reliable reasoning behaviors over long horizons.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Process Verification, Long-Horizon Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tFXBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds process rewards by checking intermediate artifacts, including citations, code execution records, image crops, and numerical equalities. A shared interface returns bounded credits assigned to interaction steps. These credits are combined with final-answer rewards for GRPO or PPO. Training starts with outcome rewards and then uses a random mixing weight shared by each prompt group. Experiments across four domains with Qwen3.5-4B and 9B report higher final accuracy than outcome-only GRPO. The aim is to improve intermediate reasoning and make agent behavior more reliable.

Critical strengths
  1. The interface is simple and practical. It separates verification rules from reward composition and policy optimization. The same HotpotQA verifier improves both GRPO and PPO by 2.45 and 2.50 percentage points, respectively (Table 3). This gives concrete support for reuse across optimizers.

  2. The evaluation covers four domains and two model sizes. Table 1 reports gains over outcome-only GRPO in all eight settings. For the 4B model, the method also exceeds the LLM-judge baseline on three of four tasks. These results provide useful evidence that the local checks can supply effective process supervision.

Critical weaknesses
  1. Using process supervision to assign RL rewards is not new. For example, VPR uses symbolic or algorithmic verifiers to produce turn-level RL rewards, and VPRMs combine rule-based process rewards with outcome rewards. Tables 1–3 show that the complete system is effective, but do not include direct comparisons with these approaches. The LLM-judge baseline is useful, but does not establish an advantage over existing verifiable process-reward designs. The contribution beyond applying process supervision through a shared interface therefore remains unclear.

  2. The evidence does not directly support the claims about long-horizon stability and self-reflection. Appendix C limits the interactive tasks to 3–5 turns. The main results measure final accuracy, and Figure 4 shows training accuracy. Neither directly measures reasoning consistency or correction behavior. Section 3.4 defines a joint outcome-and-consistency metric, but no results for it are reported. The reported gains support improved task performance in these settings, while the broader claims remain untested.

Questions for authors
  1. What is the main technical advance over existing verifiable process-reward methods such as VPR and VPRMs? Could the authors provide a comparison in a compatible setting, using the same backbone, training budget, and available verification information, or explain why these methods are not directly comparable? This would help determine whether the gains come from the proposed design or from adding process supervision itself.

  2. On the existing evaluation trajectories, how does the joint outcome-and-consistency metric in Equation (9) compare with outcome-only GRPO? Please use the same checks and evidence requirements for both policies. This would directly test whether higher final accuracy also comes with more consistent intermediate work.

Reviewer_ZWmwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a plug-and-play verification framework for improving agentic reinforcement learning through explicit process rewards. The framework decouples intermediate artifact verification from reward composition and policy optimization. It employs domain-specific deterministic verifiers to assess intermediate artifacts, such as retrieved evidence, numerical equalities, code execution records, and visual observations. The resulting verification scores are attributed to their source interaction steps through a unified interface and combined with final outcome rewards for policy optimization.

Critical strengths
  1. A practical and modular framework for integrating process verification into agentic RL.
  2. Consistent improvements over GRPO.
Critical weaknesses
  1. Insufficient comparison with existing process-reward and fine-grained credit assignment methods.
  2. Limited experimental evidence for the claimed stabilization of long-horizon agentic reasoning.
Questions for authors
  1. How does the proposed framework compare with existing process-reward methods?
  2. What evidence directly supports the claim of improved long-horizon reasoning stability?
  3. How reliably do local verification scores reflect useful reasoning progress?
Reviewer_RiftOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a plug-and-play framework that uses deterministic domain-specific verifiers to give step-level process rewards during RL. Across four benchmarks and two Qwen3.5 model sizes, it consistently improves over outcome-only GRPO, with supporting PPO and reward ablations.

Critical strengths
  1. The author cleanly separates deterministic process verification from RL optimization, giving reusable step-level rewards without relying on learned critics or LLM judges.

  2. Their method consistently improves over outcome-only RL across multiple tasks, model sizes, and both GRPO and PPO, while also providing useful learning signals when outcome rewards are tied.

Critical weaknesses
  1. The author claims improved long-horizon stability and self-reflection, but the experiments mostly use short 3–5 turn settings and report only final accuracy, without directly measuring stability, consistency, or long-horizon credit assignment.

  2. The empirical evidence is not strong enough to isolate the method: results use a single seed, gains are often small, key baselines and verifier ablations are missing, and some implementation details needed to reproduce the verifier and reward setup are unclear.

Questions for authors
  • Are the reported gains robust across multiple seeds, and do they still hold when the baselines use the same certificate format and prompting?

  • What evidence supports the claims about long-horizon stability and self-reflection, given that the current tasks are mostly short-horizon and these behaviors are not directly measured?

  • Can the authors isolate what is actually driving the gains with stronger ablations, including verifier-only, fixed vs. random reward mixing, random/format-only rewards, and comparisons to closely related methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.