acceptodds
Under review as a conference paper at ICLR 2027

LLM Value Functions for Credit Assignment in Agentic RL

Abstract

Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.

Primary area:
reinforcement learning
Keywords:
Large language models, Reinforcement learning, Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QK6mOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses credit assignment in agentic RL using a frozen LLM as the critic model in two training settings: LVRL and LVOPD. In the LVRL setting, consecutive state-value differences predicted by the critic serve as step-level advantages, while anchoring the initial value to group success rate and the terminal value to actual reward to ensure that these advantages sum to the trajectory’s group-relative outcome. In the LVOPD setting, the same advantages selectively gate teacher distillation, allowing useful student behavior to differ from the teacher. Across AppWorld, BabyAI, and SWE-bench Verified, LVRL outperforms GRPO and PPO. LVOPD improves AppWorld results compared to two distillation baselines. Further Monte-Carlo analysis show agreement with the critic's predicted directions.

Critical strengths

Eeffective learning under sparse-reward cold starts. By drawing on a frozen LLM’s semantic knowledge, LVRL produces nonzero step-level advantages even when all rollouts fail, allowing useful intermediate actions to be reinforced while preserving the trajectory-level advantage sum.

Critical weaknesses
  1. The experiments do not isolate the proposed innovations from stronger supervision by the critic model.
  • Lack of a baseline using the same frozen model (e.g, opus-4.6) for simpler step scoring or progress estimation.
  • The ablation in Table 3 removes peer context and anchoring together, leaving their individual contributions unresolved.
  • To substantiate the claimed efficiency advantage, the paper should at least report total API costs (summarization costs included), GPU-hours, wall-clock training time, and the total cost required to reach the same performance level. Ideally, the baselines should also be allowed to spend an equivalent additional budget on more rollouts to ensure the same inference budgets between different methods.
  • The paper should also include one or two representative baselines from the most closely related step-level credit assignment methods (e.g, [1-2]) that are suitable for these environments. Outperforming GRPO/PPO alone is insufficient to fully establish the method’s relative advantages.
  1. The experiments of LVOPD does not consider the changes to the training objective. Comparison with Pure OPD adds an RL objective, while comparison with trajectory-gated OPD changes both the RL advantages and the distillation gate. Thus, a fixed LVRL objective with uniform, trajectory-level, proposed step-level, and random gates shoud be compared.

References:

  1. Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.
  2. Liu, Xiaoqian, et al. "Agentic reinforcement learning with implicit step rewards." International Conference on Learning Representations. 2026.
Questions for authors

See weaknesses above.

7 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

Q1: LLM-as-a-Judge baseline with the same frozen model

Lack of a baseline using the same frozen model (e.g, opus-4.6) for simpler step scoring or progress estimation.

A1:

We added this baseline on AppWorld (Qwen3-8B, Test-Challenge), with everything except the credit mechanism identical to LVRL: same Claude Opus 4.6, same rollout and update budget, and same evaluation protocol.

  • What the judge sees: the full trajectory and its final outcome (success or failure), exactly like LVRL's critic, but no peer rollouts from the current policy.
  • What it outputs: a score in [−1, 1] for every action, one query per trajectory.
  • How scores become advantages: each raw judge score s∈[−1,1]s \in [-1, 1] is scaled into a step reward via α⋅s\alpha \cdot s with α=0.1\alpha = 0.1 (fixed a priori following standard reward-shaping magnitude controls, without test tuning). The outcome reward is added at the final step, and advantages follow GRPO process supervision (z-score within the group, then return-to-go).
AppWorld Test-Challenge, Qwen3-8BSuccess (%)
GRPO19.8 ± 3.8
LLM step-reward judge (same Opus 4.6)19.3 ± 1.1†
LVRL26.3 ± 0.5

†1 seed; ± is the std over 3 evaluation runs.

As shown above, the same frontier model, at the same budget, gives no gain over GRPO when used as a conventional step judge. Thus, stronger supervision alone does not explain LVRL's gains. Training logs point to two failure modes that align with the components:

  1. Uncalibrated positive bias. Scoring actions in isolation without policy-conditioned peer context produces positively skewed scores (mean +0.30 on a [−1, 1] scale), rather than measuring progress relative to what the current policy typically achieves.
  2. Unconstrained reward injection. Adding step scores on top of sparse outcomes distorts trajectory-level return magnitudes and leaves 1.8% of failed trajectories with net-positive credit. By contrast, LVRL's unnormalized anchored value differences telescope to R(τ)−μGR(\tau) - \mu_G, strictly confining the critic to within-trajectory credit redistribution.

Finally, we would like to emphasize that LVRL's gains are not tied to one proprietary critic: on AppWorld (Qwen3-4B, Table 4), replacing Opus 4.6 with GLM-5.2 still reaches 22.3% (vs. 10.7% GRPO in Table 1 and 24.3% Opus 4.6 LVRL).

AuthorsOpenReview ↗

Rebuttal (Part 2)

Q2: Separate ablations of group-conditioned calibration and endpoint anchoring

The ablation in Table 3 removes peer context and anchoring together, leaving their individual contributions unresolved.

A2:

We ablated each component separately on BabyAI (Qwen3-8B). We chose BabyAI because the frozen critic's signal is essential here to overcome the cold-start bottleneck where GRPO completely fails (reaching only 2.7% ID / 0.0% OOD). We only run each ablation once due to limited time.

VariantIDOOD
Full LVRL75.0 ± 2.065.5 ± 3.0
w/o peer rollouts (keeps μG\mu_G and outcome anchors; drops peer summaries)67.245.8
w/o anchoring (keeps peer summaries; drops μG\mu_G and outcome anchors)71.260.4
  • Policy-conditioned peer context matters most for generalization: Removing peer rollouts causes a severe drop in out-of-distribution transfer (down by 7.8% ID and 19.7% OOD), showing that in-context calibration to the current policy's behavior is critical when navigating unseen layouts.
  • Endpoint anchoring ensures stable credit assignment: Removing the μG\mu_G and terminal outcome anchors also degrades performance (down by 3.8% ID and 5.1% OOD), as unanchored values introduce unconstrained return drift across episodes.

We will expand Table 3 in the revised manuscript to report these separate ablations across environments.

AuthorsOpenReview ↗

Rebuttal (Part 3)

Q3: Cost and efficency

To substantiate the claimed efficiency advantage, the paper should at least report total API costs (summarization costs included), GPU-hours, wall-clock training time, and the total cost required to reach the same performance level. Ideally, the baselines should also be allowed to spend an equivalent additional budget on more rollouts to ensure the same inference budgets between different methods.

A3:

  • API cost (summarization included): about $0.12 per AppWorld trajectory (Table 6, Appendix B.1), i.e. ~$4.0k per 500-update run (32,000 trajectories), and about $0.21 per BabyAI trajectory, i.e. ~$2.1k per 150-update run (9,600 trajectories), at Claude Opus 4.6 list prices with no caching or batch discounts.
  • GPU-hours and wall-clock (measured on our rebuttal reruns, B200):
SettingHardwareMin / updateUpdatesWall-clockGPU-hours
AppWorld, Qwen3-8B1 node × 8 GPUs12.5–14.7500104–122 h830–980
BabyAI, Qwen3-8B2 nodes × 8 GPUs~31150~78 h~1,250

Rollout generation and policy updates dominate each iteration and are identical between GRPO and LVRL (PPO additionally trains a critic head). LVRL adds no GPU compute; its only overhead is the parallelized critic phase after each rollout: 2 min of a 30-min BabyAI update (≈7%) without rate-limiting, rising to 6 min (≈20%) only when a shared API key was throttled.

  • Cost to reach the same performance (Fig. 9, training reward):
    • AppWorld (Qwen3-4B): GRPO ends at 0.68 after 500 updates. LVRL passes that level after 100–120 updates—using 4× less GPU time (critic overhead included) at $0.8–1.0k in undiscounted API cost.
    • BabyAI (Qwen3-4B): LVRL passes 0.9 by update ~45, while GRPO and PPO stay near 0 for ~140 of the 150 updates and never reach that level within the budget.
  • Giving GRPO an equal budget of extra rollouts:
    • Fig. 9 already shows the rollout-budget axis: GRPO spends 4–5× more rollouts to reach the training reward LVRL reaches in ~100 updates.
    • On AppWorld, GRPO's training reward is nearly flat over its last ~100 updates (Fig. 9), so additional rollouts at the end yield diminishing returns.
    • Finally, the cost is not tied to Claude: with open-weights GLM-5.2 as the critic, LVRL still reaches 22.3 ± 4.1% vs. GRPO's 10.7 ± 2.8% on Qwen3-4B (Tables 1 and 4).

We will add per-method GPU-hours, wall-clock time, and total API cost for all runs to the appendix.

Reviewer_ESpsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LVRL, which leverages a frozen strong LLM to estimate intermediate state values using completed trajectories and policy-specific rollout context, enabling step-level credit assignment through consecutive value differences, and further extends this approach to selective on-policy distillation, achieving superior performance over GRPO and PPO across multiple agentic RL benchmarks.

Critical strengths
  1. The paper proposes an effective LLM-based approach to step-level credit assignment for agentic RL, achieving substantial improvements over GRPO and PPO across multiple benchmarks.

  2. The methodology and experimental settings are clearly described, with detailed implementation information that facilitates reproducibility.

Critical weaknesses
  1. The experiments primarily compare LVRL against GRPO and PPO, while omitting more recent step-level credit assignment methods such as GiGPO, SALT, and VinePPO. These comparisons would help better establish the effectiveness of LVRL relative to existing approaches.

  2. Although LVRL achieves substantial improvements on BabyAI, it remains unclear whether these gains primarily come from policy-aware value estimation or simply from the strong LLM critic providing semantic progress signals similar to task-specific process rewards. Since BabyAI is a relatively simple environment, a strong LLM may already identify useful intermediate actions without policy-specific calibration. The current ablation studies do not sufficiently disentangle these factors.

Questions for authors
  1. Why are recent step-level credit assignment methods, such as GiGPO, SALT, and VinePPO, not included as baselines? How does LVRL compare with these methods in terms of performance and computational cost?

  2. How can the authors distinguish the benefits of policy-aware value estimation from those of generic semantic progress estimation by a strong LLM? Would a simpler LLM-based process reward baseline achieve comparable performance, particularly on BabyAI?

  3. Could the authors provide separate ablations for group-conditioned calibration and endpoint anchoring? The current ablation removes multiple components simultaneously, making their individual contributions difficult to assess.

5 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

Q1: Compare step-level credit-assignment baselines

Why are recent step-level credit assignment methods, such as GiGPO, SALT, and VinePPO, not included as baselines? How does LVRL compare with these methods in terms of performance and computational cost?

A1:

We added GiGPO on AppWorld (Qwen3-8B, Test-Challenge) with its official hyperparameters (γ=0.95,ω=1\gamma = 0.95, \omega = 1) and the same rollout budget and evaluation protocol as all other methods. We run three different seeds and report the mean and std across them.

How we define identical states. In AppWorld, the stateful REPL and database before step tt are uniquely determined by the executed code prefix. Two steps therefore share an anchor state if and only if they belong to the same task group and their sequences of executed code blocks {a0…at−1}\{a_0 \dots a_{t-1}\} match (ignoring comments and whitespace, etc), faithfully preserving GiGPO's exact-state-matching design without introducing an ad-hoc semantic matcher. The initial state is trivially shared and is excluded from the coverage numbers below.

The final results are shown below:

MethodAppWorld Test-Challenge (Qwen3-8B)
GRPO19.8±3.8%19.8 \pm 3.8\%
GiGPO (with official γ=0.95,ω=1\gamma = 0.95, \omega = 1)13.3±0.9%13.3 \pm 0.9\%
LVRL (Ours)26.3±0.5%26.3 \pm 0.5\%

Under this implementation, we found two issues that can potentially explain this gap:

  • Exact state matches are rare. Only 11–12% of non-initial steps share an anchor state with another rollout, so only about 10% of steps ever receive a non-zero step advantage. Notably, iStar [1] itself identifies sparse state overlap as a known failure mode of GiGPO.
  • The few matches yield noisy return estimates. Across all steps, anchor groups average only 1.15 states—meaning almost every step is a singleton, and the rare matched steps typically pair only a handful of trajectories. Averaging terminal returns over such tiny groups produces high-variance step advantages. Adding these noisy terms (with weight ω=1\omega = 1) into the trajectory advantage injects high-variance noise into an otherwise stable GRPO gradient, explaining the drop below GRPO.

This reflects a structural limitation of matching identical states in open-ended, code-action environments. SALT relies on the same principle (a trajectory graph over shared states). VinePPO estimates values from Monte-Carlo continuations at intermediate states, which also requires full environment replay in stateful AppWorld. Even with fewer continuations per state, performing state-level rollouts and stateful replays at every training batch multiplies environment interaction by an order of magnitude.

On BabyAI, these methods face an even more basic bottleneck: the base policies start near 0% success, so almost every rollout group has all-zero rewards. Methods that aggregate empirical outcome returns—whether over shared states (GiGPO, SALT) or over MC continuations (VinePPO) receive essentially no signal and stall at the same cold start as GRPO.

[1] Liu, Xiaoqian, et al. "Agentic reinforcement learning with implicit step rewards." International Conference on Learning Representations. 2026.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Q2: LLM-as-a-Judge baseline with the same frozen model

How can the authors distinguish the benefits of policy-aware value estimation from those of generic semantic progress estimation by a strong LLM? Would a simpler LLM-based process reward baseline achieve comparable performance, particularly on BabyAI?

A2:

We added this baseline on AppWorld (Qwen3-8B, Test-Challenge), with everything except the credit mechanism identical to LVRL: same Claude Opus 4.6, same rollout and update budget, and same evaluation protocol.

  • What the judge sees: the full trajectory and its final outcome (success or failure), exactly like LVRL's critic, but no peer rollouts from the current policy.
  • What it outputs: a score in [−1, 1] for every action, one query per trajectory.
  • How scores become advantages: each raw judge score s∈[−1,1]s \in [-1, 1] is scaled into a step reward via α⋅s\alpha \cdot s with α=0.1\alpha = 0.1 (fixed a priori following standard reward-shaping magnitude controls, without test tuning). The outcome reward is added at the final step, and advantages follow GRPO process supervision (z-score within the group, then return-to-go).
AppWorld Test-Challenge, Qwen3-8BSuccess (%)
GRPO19.8 ± 3.8
LLM step-reward judge (same Opus 4.6)19.3 ± 1.1†
LVRL26.3 ± 0.5

†1 seed; ± is the std over 3 evaluation runs.

As shown above, the same frontier model, at the same budget, gives no gain over GRPO when used as a conventional step judge. Thus, stronger supervision alone does not explain LVRL's gains. Training logs point to two failure modes that align with the components:

  1. Uncalibrated positive bias. Scoring actions in isolation without policy-conditioned peer context produces positively skewed scores (mean +0.30 on a [−1, 1] scale), rather than measuring progress relative to what the current policy typically achieves.
  2. Unconstrained reward injection. Adding step scores on top of sparse outcomes distorts trajectory-level return magnitudes and leaves 1.8% of failed trajectories with net-positive credit. By contrast, LVRL's unnormalized anchored value differences telescope to R(τ)−μGR(\tau) - \mu_G, strictly confining the critic to within-trajectory credit redistribution.

Finally, we would like to emphasize that LVRL's gains are not tied to one proprietary critic: on AppWorld (Qwen3-4B, Table 4), replacing Opus 4.6 with GLM-5.2 still reaches 22.3% (vs. 10.7% GRPO in Table 1 and 24.3% Opus 4.6 LVRL).

AuthorsOpenReview ↗

Rebuttal (Part 3)

Q3: Separate ablations of group-conditioned calibration and endpoint anchoring

Could the authors provide separate ablations for group-conditioned calibration and endpoint anchoring? The current ablation removes multiple components simultaneously, making their individual contributions difficult to assess.

A3:

We ablated each component separately on BabyAI (Qwen3-8B). We chose BabyAI because the frozen critic's signal is essential here to overcome the cold-start bottleneck where GRPO completely fails (reaching only 2.7% ID / 0.0% OOD). We only run each ablation once due to limited time.

VariantIDOOD
Full LVRL75.0 ± 2.065.5 ± 3.0
w/o peer rollouts (keeps μG\mu_G and outcome anchors; drops peer summaries)67.245.8
w/o anchoring (keeps peer summaries; drops μG\mu_G and outcome anchors)71.260.4
  • Policy-conditioned peer context matters most for generalization: Removing peer rollouts causes a severe drop in out-of-distribution transfer (down by 7.8% ID and 19.7% OOD), showing that in-context calibration to the current policy's behavior is critical when navigating unseen layouts.
  • Endpoint anchoring ensures stable credit assignment: Removing the μG\mu_G and terminal outcome anchors also degrades performance (down by 3.8% ID and 5.1% OOD), as unanchored values introduce unconstrained return drift across episodes.

We will expand Table 3 in the revised manuscript to report these separate ablations across environments.

Reviewer_9VcTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LVRL, which asks a frozen LLM critic to retrospectively estimate state values along agent trajectories and uses consecutive value differences for credit assignment. LVOPD further uses negative value changes to decide where teacher-distribution distillation should be applied. The method is simple and reports promising results across several agent environments.

Critical strengths
  • LVRL addresses the important problem of sparse rewards in long-horizon agent tasks without training a separate value model.
  • Reusing the same critic signal for policy optimization and selective distillation is an elegant and practical design.
  • The experiments across multiple environments suggest that semantic trajectory evaluation by an LLM can provide useful training signals.
Critical weaknesses
  1. The reliability of the elicited state values is not sufficiently established. The method asks a general-purpose LLM, without value-function training, to produce abstract numerical state estimates. The results show that the signal can be useful, but do not establish whether these numbers are reliable value estimates or whether their magnitudes are meaningful for credit assignment.

  2. A natural LLM-generated reward baseline is missing. An alternative is to ask the same LLM for concrete step-level rewards or action-quality judgments and derive advantages from them. Without this comparison, it is unclear whether direct value elicitation is preferable to a conventional LLM-as-a-Judge approach.

  3. The benefit of critic-based OPD selection is not demonstrated against existing fine-grained alternatives. Selective OPD methods already use evaluative signals to decide where teacher supervision should be applied, but the current experiments mainly compare against uniform or coarse trajectory-level selection. A matched comparison with a representative method such as AOPD is needed. Otherwise, LVOPD appears to be a convenient combination of LVRL and selective distillation rather than a clearly justified integration.

Questions for authors
  1. Could the authors demonstrate that using an LLM to estimate state values is more effective than using it to directly provide fine-grained rewards?

  2. Could the authors compare LVOPD with AOPD or another fine-grained selective OPD method? Please use the same RL objective, teacher, and distillation budget for a fair comparison.

4 comments
AuthorsOpenReview ↗

Rebuttal (Part 1)

Q1: Reliability of the prediction values

The reliability of the elicited state values is not sufficiently established. The method asks a general-purpose LLM, without value-function training, to produce abstract numerical state estimates. The results show that the signal can be useful, but do not establish whether these numbers are reliable value estimates or whether their magnitudes are meaningful for credit assignment.

A1:

We agree that the critic's raw outputs should not be read as calibrated success probabilities—and LVRL does not need them to be. Because LVRL normalizes step differences ΔV\Delta V by group-level σstep\sigma_{\text{step}} (Sec. 3.2), any global scale mismatch cancels out. Credit assignment only requires that (i) the sign of ΔV\Delta V is correct (Sec. 4.4), and (ii) larger predicted jumps correspond to larger real shifts in success probability (estimated by MC rollouts). We now verified (ii) directly.

We use our AppWorld Monte-Carlo dataset from Sec. 4.4 (180 trajectories, M=32M=32 rollouts per state), we drop the first and last steps so endpoint anchoring cannot affect the results (3,347 interior steps). We group steps by the size of the critic's predicted jump (∣ΔVpred∣\lvert \Delta V^{\text{pred}} \rvert) and check two things: the average real MC change (∣ΔVMC∣\lvert \Delta V^{\text{MC}} \rvert), and what percentage of steps in that bin are real turning points (a major pitfall or recovery where real MC success probability shifts by ≥0.375\ge 0.375, the 3σ3\sigma threshold in Sec. 4.4).

Predicted jump ∣ΔVpred∣\lvert \Delta V^{\text{pred}} \rvert# stepsAvg. real change ∣ΔVMC∣\lvert \Delta V^{\text{MC}} \rvert% of steps with real MC shift ≥0.375\ge 0.375
=0= 01,1620.0751.5%
(0,0.1](0, 0.1]2,0060.0772.7%
(0.1,0.25](0.1, 0.25]1390.15613.7%
>0.25> 0.25400.36245.0%

In long-horizon agent tasks, true critical turning points are extremely sparse. If the critic's magnitudes were uninformative, every bin would hover near that baseline. Instead, when the critic predicts zero change, almost none of those steps are turning points (1.5%, sitting at the M=32M=32 noise floor). But when it predicts a jump >0.25> 0.25, 45.0% of those steps are true major turning points—a massive 15× to 20× enrichment over the baseline—with real MC shifts averaging 0.362.

Furthermore, this trend is not driven by outcome leakage. Looking exclusively at failed trajectories gives the same sharp rise (1.4% → 3.5% → 25.0% → 47.0%). Even when all trajectories end in failure, larger predicted drops reliably pinpoint decisive mistakes rather than routine steps.

We will add this analysis to Section 4.4 / Appendix F in the revised paper.

AuthorsOpenReview ↗

Rebuttal (Part 2)

Q2: LLM-generated reward baseline with the same frozen model

A natural LLM-generated reward baseline is missing. An alternative is to ask the same LLM for concrete step-level rewards or action-quality judgments and derive advantages from them. Without this comparison, it is unclear whether direct value elicitation is preferable to a conventional LLM-as-a-Judge approach.

A2:

We added this baseline on AppWorld (Qwen3-8B, Test-Challenge), with everything except the credit mechanism identical to LVRL: same Claude Opus 4.6, same rollout and update budget, and same evaluation protocol.

  • What the judge sees: the full trajectory and its final outcome (success or failure), exactly like LVRL's critic, but no peer rollouts from the current policy.
  • What it outputs: a score in [−1, 1] for every action, one query per trajectory.
  • How scores become advantages: each raw judge score s∈[−1,1]s \in [-1, 1] is scaled into a step reward via α⋅s\alpha \cdot s with α=0.1\alpha = 0.1 (fixed a priori following standard reward-shaping magnitude controls, without test tuning). The outcome reward is added at the final step, and advantages follow GRPO process supervision (z-score within the group, then return-to-go).
AppWorld Test-Challenge, Qwen3-8BSuccess (%)
GRPO19.8 ± 3.8
LLM step-reward judge (same Opus 4.6)19.3 ± 1.1†
LVRL26.3 ± 0.5

†1 seed; ± is the std over 3 evaluation runs.

As shown above, the same frontier model, at the same budget, gives no gain over GRPO when used as a conventional step judge. Thus, stronger supervision alone does not explain LVRL's gains. Training logs point to two failure modes that align with the components:

  1. Uncalibrated positive bias. Scoring actions in isolation without policy-conditioned peer context produces positively skewed scores (mean +0.30 on a [−1, 1] scale), rather than measuring progress relative to what the current policy typically achieves.
  2. Unconstrained reward injection. Adding step scores on top of sparse outcomes distorts trajectory-level return magnitudes and leaves 1.8% of failed trajectories with net-positive credit. By contrast, LVRL's unnormalized anchored value differences telescope to R(τ)−μGR(\tau) - \mu_G, strictly confining the critic to within-trajectory credit redistribution.

Finally, we would like to emphasize that LVRL's gains are not tied to one proprietary critic: on AppWorld (Qwen3-4B, Table 4), replacing Opus 4.6 with GLM-5.2 still reaches 22.3% (vs. 10.7% GRPO in Table 1 and 24.3% Opus 4.6 LVRL).

AuthorsOpenReview ↗

Rebuttal (Part 3)

Q3: LVOPD vs. fine-grained selective OPD (AOPD) under a fixed LVRL objective

Could the authors compare LVOPD with AOPD or another fine-grained selective OPD method? Please use the same RL objective, teacher, and distillation budget for a fair comparison.

A3:

We ran a strictly matched comparison on AppWorld (Qwen3-4B):

  • Held fixed: the LVRL objective (same Qwen3-235B critic, same advantages, same PPO clipping), the Qwen3-235B teacher, and the Qwen3-4B student.
  • Changed: only the distillation term, swapped for AOPD (Jia et al., 2026).

We followed the official AOPD code and verified our implementation against it with unit tests: forward KL on the teacher's top-32 tokens with the student's full-vocabulary log-probabilities, τ=0\tau = 0, and the PPO-clipped OPD branch. We also ran standalone AOPD.

AppWorld, Qwen3-4BTrain (upd 1)Train (upd 10)Train (upd 50)Test-Challenge (%)
LVOPD9.5%38.7%~48% (Fig. 10, 3 seeds)20.2 ± 1.8 (Table 2)
LVRL + AOPD6.9%1.4%0% (from upd 15)0.0 ± 0.0
AOPD9.1%0.8%0% (from upd 14)0.0 ± 0.0

Both AOPD variants collapse within 15 updates and never recover through update 50, whereas LVOPD climbs steadily through 200 updates (~70% training reward, Fig. 10), confirming the collapse comes from AOPD itself:

  • Over-triggering: AOPD applies forward-KL imitation whenever the student is more confident than the teacher on a sampled token. For our student at initialization, that fires on 95% of tokens.
  • Entropy blow-up: Because forward KL spreads probability mass to cover the teacher's full distribution, applying it almost everywhere causes entropy to explode—top-16 entropy jumps from 0.2 to 1.8 (vs. 0.3 under LVOPD).
  • Behavioral collapse: Over multi-turn code generation, the policy quickly degenerates: it stops writing valid code, never calls complete_task, and solves 0 of 1,251 Test-Challenge episodes (417 tasks × 3, every run hitting the 70-turn limit).

Why LVOPD stays stable: LVOPD only distills on steps where estimated value drops (20–35% of tokens) and uses mode-seeking reverse KL, which keeps the policy sharp instead of spreading probability mass. It corrects mistakes without destabilizing the student's valid strategies (Section 3.3).

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 9VcT,

Thank you again for your detailed and constructive review! As the discussion period closes on Nov 18, we wanted to follow up on our response, which was posted late because the requested experiments took considerable time to run. We would be grateful if you could take a look and let us know whether these address your concerns. If they do, we would appreciate your considering an updated assessment.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.