acceptodds
Under review as a conference paper at ICLR 2027

Lean4AgenticRL: Advancing Agentic RL with Verifiable Multi-level Process Rewards

Abstract

Enhancing the agentic capabilities of Large Language Models (LLMs) to tackle complex software engineering (SWE) and general terminal-usage problems is a pressing challenge. Current state-of-the-art LLMs often go through large-scale agentic Reinforcement Learning (RL). However, they mostly rely on high-level outcome rewards, with minimal consideration of low-level toolcall and mid-level workflow-stage rewards, which makes the training less effective. To overcome this limitation, we propose **Lean4AgenticRL**, which, to the best of our knowledge, is the first framework that grounds multi-level process rewards in the unified Lean4 formalization schema for open-ended agentic RL. **Lean4AgenticRL** first builds a *problem-specific workflow* system that balances the agent's own capability and controllability during execution, laying the foundation for further verification. Subsequently, our workflow formalization and multi-level reward annotation methods enable agent behavior verification within a general harness system. It provides multi-level rewards to support more effective agentic RL with the workflow-stage-normalized GRPO algorithm. Using **Lean4AgenticRL**, we train **LAR-9B/27B**, a family of models with enhanced agentic capability. Extensive experiments indicate that our model outperforms the leading model of the similar size. **LAR-27B** achieves **77.3%** on TerminalBench-2.1. On average, our models outperform the base model by **9.89%** and surpass the second-best model of comparable size we know by **5.07%**. These results indicate the potential of integrating formal methods in enhancing open-ended agentic RL.

Primary area:
reinforcement learning
Keywords:
Lean, agentic system, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_QEFgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Lean4AgenticRL, a framework for assigning multi-level process rewards to open-ended agentic RL trajectories. It first has an agent explore a task and produce a problem-specific workflow, then formalizes that workflow and segments the execution trajectory into stages. The framework combines final-outcome rewards, workflow-stage rewards, and tool-call penalties, using Lean and Python checkers alongside LLM judges. The authors train LAR-9B and LAR-27B and evaluate them on SWE-Bench-Verified and TerminalBench 2.0/2.1, reporting improvements over their base models and comparable open-weight models.

Critical strengths
  1. The paper explores a potentially novel use of formal methods for agentic RL. Using Lean as part of a framework for assigning process rewards to agent trajectories is an interesting direction.

  2. The reported results are strong across several frontier coding benchmarks. LAR-9B and LAR-27B improve on their base models across SWE-Bench-Verified and TerminalBench 2.0/2.1. The paper also compares against outcome-reward GRPO and includes ablations of the workflow and tool-call penalty components.

Critical weaknesses
  1. The paper does not establish why Lean is necessary for the proposed reward checks. Many of the concrete tool-call rules, such as detecting repeated commands, no-op commands, or constant-only output, appear implementable with ordinary Python checks. The paper itself uses Python to parse commands and allows directly decidable predicates to be verified with either Lean or Python. Lean may offer advantages for composing typed contracts and checking workflow consistency, but the paper does not show that these advantages improve reward quality or downstream agent performance. A Python-only implementation of the same predicates and reward pipeline would be a key comparison.

  2. The gains cannot be cleanly attributed to the proposed Lean-based process rewards. The problem-specific workflow is generated and formalized with LLM assistance, trajectory steps are assigned to workflow stages by an LLM, and some semantic postconditions are judged by an LLM. Lean can check the formalized predicates and their composition, but it cannot guarantee that the generated workflow, stage assignments, or semantic judgments correctly capture the task. This leaves a gap between the paper’s emphasis on Lean's verifiable rewards and their actual reliability and contribution to downstream performance.

  3. The comparison with outcome-only GRPO is not a fully controlled test of the method’s benefit. The paper reports different training configurations for the proposed method and the direct GRPO baseline, including different group/batch sizes and training-step budgets. The training curves are encouraging, but without matched-compute or matched-rollout comparisons, it is difficult to determine how much of the advantage comes from the reward design rather than differences in training scale or configuration.

Questions for authors
  1. Which parts of the reward pipeline are checked by Lean, which by Python, and which by an LLM judge? Can you quantify this split and report the reliability of each component?
  2. How does performance change when the Lean checks are replaced with Python implementations of the same predicates, while keeping the workflows, reward values, and RL setup fixed?
  3. Can you provide a matched-compute or matched-rollout comparison against outcome-only GRPO, and ablations that separately test β rewards and workflow-stage normalization?
Reviewer_axPPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends recent work on Lean4Agent, a framework for formal verification of fixed-agent workflows, to open-ended coding tasks via a multi-level reward mechanism. The approach is evaluated on three coding benchmarks extending GRPO and compared to various open-weight model baselines, showing improvements across two size regimes.

Critical strengths

Overall, the approach is thoroughly formalized, well elaborated, and easy to follow. It integrates a formal verification framework to improve automated reward design and generalizes the approach from task-specific setups to open-ended tasks, which represents a crucial current area of research. The implementation is attentive to common reward-hacking indicators and integrates them into the low-level reward. Despite the fact that these indicators currently seem handpicked, they provide a crucial mitigation strategy against reward hacking. Although the evaluation is limited to coding task benchmarks, the authors provide a broad baseline coverage and consider two different size regimes. Additional detailed ablations (specifically dropping the tool call reward and the problem-specific workflow) provide evidence of the importance of all parts of the proposed approach, while a detailed case study, mostly deferred to the appendix due to space restrictions, demonstrates the usefulness of the proposed multi-level reward.

Critical weaknesses

Overall, the central contribution seems more engineering-focused, assembling the system that enables multi-level reward training while building on and reusing prior work. In particular, the annotation and reward system appears closely related to, and only incrementally adapted from, Lean4Agent.

Given that the approach crucially relies on LLM feedback, a discussion of the key implications and potential drawbacks of such a system would be important.

Finally, the reported results seem more like ablations (GRPO without the multi-level reward, Qwen without task-specific training). While these provide important evidence for the improvements achieved by the proposed approach, comparisons to similar training frameworks (beyond reporting performance for non-retrained models), e.g., Lean4Agent or other related RLVR or AI-as-a-judge approaches, would have been helpful.

Questions for authors
  • Could the authors clarify what they refer to by "verifiable" and motivate why that should be considered important
  • How much do the reported results depend on the features and capabilities of the used agent harness vs. the underlying LLM?
Reviewer_vyAUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This research article proposes Lean4AgenticRL which is a framework that integrates formal verification with multi-level process reward to improve the reinforcement learning for LLM based agents. The proposed method combines high level outcome rewards, workflow stage rewards, and tool call penalties within a unified Lean4based verification frameworks. To improve the credit assignment during training, the authors also have introduced workflow stage normalized GRPO.

The proposed system is evaluated using LAR-9B and LAR-27B models on SWE-Bench-verified and TerminalBench-2.0/2.1. The results have shown an improvement over the corresponding base models and competitive baselines. LAR-27B has achieved 77.3% on TerminalBench-2.1 whereas further experiments and ablation studies provide an evidence supporting the usefulness of this proposed methodology.

Critical strengths
  1. Novel integration of formal verification and multi level rewards The main strength/contribution of this research work is the integration of outcome level, workflow-stage, and tool call rewards within unified verification framework. I believe this not only provides a structured approach in addressing credit assignment in long horizon agentic tasks but also represents a useful and meaningful contribution beyond conventional outcome based RL.

  2. Strong experimental performance and practical relevance The evaluation of proposed framework includes various benchmarks and model sizes and demonstrates significant improvement over competitive baselines. The performance of LAR-27B reported on TerminalBench-2.1 is particularly encouraging. Additional ablation experiments provide a supporting evidence of improvements in agent performance and reduction in undesirable behaviors.

Critical weaknesses
  1. Although the proposed framework emphasizes formal verification, several important stages that includes workflow formalization, segmentation, and semantic predicate evaluation, but it still depends on LLM judgements. Appendix D2 reports an annotation failure rate of 8.03%, 6.21%, and 4.84% across various verification components. These errors might influence reward reliability and training outcomes. A clear discussion on how this annotation error propagate through the learning process would strengthen the article.

  2. The ablation study investigates the removal of problem specific workflows and tool-call penalties but it do not comprehensively isolate the contribution of workflow-stage rewards or stage-based normalizaiton. Eventually, the relative importance of these components remains unclear. Furthermore, huge training resources required raise questions about computational efficiency and accessibility. A more detailed cost-benefit analysis would improve the practical significance of the study.

Questions for authors
  1. How sensitive is the final model performance to errors in LLM-based workflow annotation and verification? Have you investigated/experimented if reducing annotation errors leads to measurable performance improvements or not.

  2. What is additional computational overhead introduced by workflow formalization and multi-level reward annotation compared with conventional GRPO training? Could you provide a comparison based on total computational cost instead of training steps alone.

Reviewer_eFLtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Lean4AgenticRL, a framework for training LLM agents with multi-level rewards. It builds a problem-specific workflows in Lean and maps execution steps to workflow stages. It combines outcome rewards, stage-level rewards and penalties for repetitive or ineffective tool calls. The authors further introduce workflow-stage-normalized GRPO. Experiments on SWE-Bench-Verified and TerminalBench report performance improvements for the resulting 9B and 27B models over their respective base models.

Critical strengths

This paper addresses the difficulty of providing informative feedback for long-horizon agent tasks. Using workflow stages as an intermediate unit of supervision provides an intuitive approach to organize process rewards and align trajectories with different execution paths.

Critical weaknesses
  1. The training comparison is mainly with outcome-only GRPO. I think comparison with other process-reward methods is still needed, especially the process-reward version of OpenClaw-RL.
  2. The ablation experiments do not separately check stage-level rewards and stage normalization. It's hard to say which part actually bring the gains.
  3. Workflow formalization, segmentation, and semantic judgments still rely on LLMs. The paper mentions this, but I am still not clear what extra benefit Lean gives over a similar structured reward pipeline.
  4. Fewer training steps does not necessarily mean less training time or lower cost. The proposed method also needs workflow formalization, trajectory segmentation, and LLM-based reward annotation, which add extra overhead. It would be useful to compare the total time and cost needed by this method and outcome-only GRPO to reach similar performance.
  5. The novelty seems limited. The method builds on Lean4Agent and existing process-reward methods. It adds task-specific workflows and stage-based reward normalization. These seem to be incremental changes.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.