- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Lean4AgenticRL, a framework for assigning multi-level process rewards to open-ended agentic RL trajectories. It first has an agent explore a task and produce a problem-specific workflow, then formalizes that workflow and segments the execution trajectory into stages. The framework combines final-outcome rewards, workflow-stage rewards, and tool-call penalties, using Lean and Python checkers alongside LLM judges. The authors train LAR-9B and LAR-27B and evaluate them on SWE-Bench-Verified and TerminalBench 2.0/2.1, reporting improvements over their base models and comparable open-weight models.
Critical strengths
-
The paper explores a potentially novel use of formal methods for agentic RL. Using Lean as part of a framework for assigning process rewards to agent trajectories is an interesting direction.
-
The reported results are strong across several frontier coding benchmarks. LAR-9B and LAR-27B improve on their base models across SWE-Bench-Verified and TerminalBench 2.0/2.1. The paper also compares against outcome-reward GRPO and includes ablations of the workflow and tool-call penalty components.
Critical weaknesses
-
The paper does not establish why Lean is necessary for the proposed reward checks. Many of the concrete tool-call rules, such as detecting repeated commands, no-op commands, or constant-only output, appear implementable with ordinary Python checks. The paper itself uses Python to parse commands and allows directly decidable predicates to be verified with either Lean or Python. Lean may offer advantages for composing typed contracts and checking workflow consistency, but the paper does not show that these advantages improve reward quality or downstream agent performance. A Python-only implementation of the same predicates and reward pipeline would be a key comparison.
-
The gains cannot be cleanly attributed to the proposed Lean-based process rewards. The problem-specific workflow is generated and formalized with LLM assistance, trajectory steps are assigned to workflow stages by an LLM, and some semantic postconditions are judged by an LLM. Lean can check the formalized predicates and their composition, but it cannot guarantee that the generated workflow, stage assignments, or semantic judgments correctly capture the task. This leaves a gap between the paper’s emphasis on Lean's verifiable rewards and their actual reliability and contribution to downstream performance.
-
The comparison with outcome-only GRPO is not a fully controlled test of the method’s benefit. The paper reports different training configurations for the proposed method and the direct GRPO baseline, including different group/batch sizes and training-step budgets. The training curves are encouraging, but without matched-compute or matched-rollout comparisons, it is difficult to determine how much of the advantage comes from the reward design rather than differences in training scale or configuration.
Questions for authors
- Which parts of the reward pipeline are checked by Lean, which by Python, and which by an LLM judge? Can you quantify this split and report the reliability of each component?
- How does performance change when the Lean checks are replaced with Python implementations of the same predicates, while keeping the workflows, reward values, and RL setup fixed?
- Can you provide a matched-compute or matched-rollout comparison against outcome-only GRPO, and ablations that separately test β rewards and workflow-stage normalization?