acceptodds
Under review as a conference paper at ICLR 2027

Gated Successor Calibration for Solver-Guided LLM Agents

Abstract

Game solvers provide turn-level feedback for training LLM agents when task outcomes are sparse. However, a reduction in solver cost does not reveal whether the chosen action is better than the alternatives available in the same state. Solver feedback can also continue to shape updates when every rollout in a sampled task group succeeds. We propose Gated Successor Calibration (GSC), a method that calibrates solver feedback against local alternatives and gates its use by group success. Given a state, GSC subtracts the mean solver progress of distinct effective successors from the chosen action's progress. Intuitively, the same progress deserves more credit when alternative actions make less progress than when they make comparable progress. For all-success groups, GSC removes the solver-derived process term while leaving outcome credit and the policy objective unchanged. Across Sokoban, Rush Hour, and Minesweeper, GSC outperforms a strong solver-guided reinforcement learning baseline and also exhibits zero-shot transfer to ALFWorld and WebShop when trained only on game tasks. Specifically, with Qwen3-4B-Instruct-2507, GSC achieves average in-domain success and overall zero-shot success 6.8 and 1.1 percentage points higher than this baseline, respectively.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, credit assignment, solver-guided learning, process supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gPvWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gated Successor Calibration (GSC) to improve solver-guided reinforcement learning for LLM agents. Building on CAST, the method introduces two components: Successor Centering, which adjusts solver-based progress rewards by comparing them with the average progress of alternative successor states, and All-Success Gating, which removes process-level rewards when all sampled trajectories in a group succeed. The authors evaluate GSC on three puzzle environments and further test its zero-shot generalization on ALFWorld and WebShop. Compared to CAST, GSC improves average success rates both on the training domains and unseen difficulty levels.

Critical strengths
  1. The paper addresses an important problem in credit assignment for LLM agents. Comparing solver feedback against alternative actions is a reasonable way to distinguish meaningful progress from plain actions.
  2. The experimental results are promising and consistent across multiple environments. GSC outperforms the strong CAST baseline across all three games, with improvements on both in-domain and unseen-difficulty tasks.
Critical weaknesses
  1. The idea of this paper is not novel, and the analysis of the proposed components is insufficient. Progress-based rewards for credit assignment have already been explored in prior work, and GSC mainly introduces two relatively simple modifications to existing solver-guided RL. More importantly, the ablation results show that neither component alone improves average in-domain performance over CAST, while combining them leads to a substantial 6.8 percentage-point gain. This interaction is interesting but not sufficiently explained. The paper would benefit from a deeper analysis of why the individual components fail to improve performance but work so well together.

  2. The generalization improvement is limited. The zero-shot transfer gains on ALFWorld and WebShop are marginal compared to CAST, with an average improvement of only 1.1 percentage points. Moreover, the paper does not report error bars or variance across runs, making it difficult to determine whether these improvements are statistically significant.

Questions for authors
  1. Could the authors provide more insight into why the two proposed components work well together but not individually?
  2. Could the author provide some discussion about why their methods novel compared with prior methods using progressive reward? and why it's worth to work on games using this method.
  3. Could the authors provide results across multiple random seeds, particularly for the zero-shot transfer experiments? The purpose is to make sure the improvement is clean and significant.
Reviewer_LvyROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In solver-guided reinforcement learning for large language model agents, each turn is rewarded by a reduction in the remaining cost of the game solver. However, this says nothing about alternatives in the same state, and the process reward continues to contribute even when every rollout of a task succeeds. The paper modifies this process term by subtracting the mean progress over all distinct effective successor states and switching the term off for task groups in which all rollouts succeed. The outcome advantage and objective remain unchanged. The paper claims that this method outperforms the solver-guided CAST baseline in-domain and on harder puzzles and transfers slightly better to two unrelated interactive benchmarks. Ablation studies demonstrate that both the all-success gating and the successor centering contribute to the gains, which are most prominent when both are used.

Critical strengths
  1. A straightforward drop-in on top of the CAST method, with both components (successor centering and all-success gating) ablated on all three games. Same-step comparisons were made of these GSC variants.

  2. The method is a reward adjustment of the CAST method, with a well-motivated design that successfully improves the original method in the reported comparison, when both components are active.

  3. The paper openly discusses training dynamics and results, addressing mid-training regressions (e.g., no-gate minesweeper) and disclosing that the GRPO/GSPO/DAPO/GiGPO numbers are three-run final checkpoint means taken from the CAST paper, whereas GSC uses selected checkpoints (see App. A.1) and reporting final step numbers next to the selected ones.

Critical weaknesses
  1. Eval Protocol: The CAST row in Table 1 is exactly CAST's published number. The CAST paper defines this number as 'the mean over three independent runs, taking each run's final checkpoint.' In contrast, GSC is one run and a selected checkpoint. The selection process is not clearly stated in the manuscript (Fig.4: 'stars mark selected checkpoints, not necessarily curve maxima'). Furthermore, Section 4.1 claims that CAST was reevaluated from an 'author-provided best checkpoint' (L247), yet the paper's own CAST run is reported to end at 46.1 (L381) on Minesweeper, not 44.7, as reported in Table 1.

  2. In an all-success group, the outcome advantage is zero and the paper confirms the final advantages of such a group are zero, i.e. no gradient from that task. This is DAPO's dynamic-sampling filter, restricted here to the all-correct case. DAPO is a baseline in Table 1. However, the relationship is not discussed, and simply dropping these groups like DAPO is not compared. Additionally, all-fail groups have a zero outcome advantage but retain the process term. Discussing this would strengthen the argument. In GSC, solved tasks still carry a signal because the process term distinguishes between efficient and less efficient solutions. Gating discards this term, but nothing shows what that costs (e.g. steps-to-solve). The only evidence I found for the gate is the minesweeper mid-training regression.

  3. Given a solver over all successors, a meaningful additional baseline is to use the solver's best successor directly, for example, by imitation or distillation. I do not think this is strictly necessary, but would strengthen the argument. Furthermore, the cost of successor enumeration, which implies a solver call for every legal successor at every turn, and the 'strong solver' for Minesweeper are not reported.

  4. Single seed per run results in no error bars, which may place the reported zero-shot gains within the margin of error.

Questions for authors
  1. How were the selected checkpoints chosen and based on what data?
  2. Is the CAST row in Table 1 your local reevaluation or the published three-run mean from CAST?
  3. How does all-success gating relate to DAPO's dynamic sampling? Why is the process term kept for all-fail groups?
  4. Does gating reduce solution efficiency on solved tasks? E.g. in terms of steps-to-solve or solution length with and without the gate.
  5. What is the per-step overhead of successor enumeration?
  6. Do you have seed results for GSC and the two ablations?
Reviewer_CjyfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Building upon the solver-guided reinforcement learning framework of CAST, this paper introduces two additional components: Successor Centering, which compares the solver-estimated progress of an action against the average progress across distinct valid successor states from the current state; and All-Success Gating, which disables process supervision when all sampled trajectories for a given instance succeed. The authors evaluate the proposed method across three game environments as well as cross-domain transfer benchmarks.

Critical strengths

The proposed approach is concise and builds directly upon the strong CAST baseline. The full configuration achieves consistent improvements across both in-distribution (ID) and unseen (OOD) test splits on all three games, indicating that the performance gains are not confined to a single game environment or difficulty level.

Critical weaknesses

Weakness 1

The paper conducts training exclusively with a single backbone model (Qwen3-4B-Instruct-2507) across three game environments (Sokoban, Rush Hour, and Minesweeper). Although the evaluation includes harder test splits and cross-domain transfer tasks, this setup is insufficient to establish whether the method generalizes across different model scales, model families, and environment characteristics. In particular, because the method relies on enumerating valid successor states and querying game-specific solvers, factors such as branching factors, solver fidelity, and cost definitions across environments can substantially affect the centered process signal, which the current experiments explore only to a limited extent.

More critically, the paper does not clearly report means, standard deviations, or confidence intervals across multiple independent training seeds for the full GSC framework and its key ablations. The reported Avg@4 metric only reflects evaluation-time sampling variance from a single checkpoint per problem, rather than capturing the stochasticity of the RL training process itself. Furthermore, the comparisons mix cited baseline numbers, an author-provided best checkpoint for CAST, and custom-selected checkpoints for GSC, requiring stricter alignment in both statistical reporting and checkpoint selection protocols. While the authors provide partial comparisons at matched training steps and at the end of training, these do not substitute for repeated independent training runs under a unified protocol. I recommend re-running CAST, full GSC, and the two component ablations across multiple independent training seeds with clearly aligned checkpoint selection criteria. In addition, given that the overall cross-domain transfer gain is only approximately 1.11.1 percentage points, rigorous uncertainty quantification is especially necessary to establish statistical significance.

Weakness 2

In Table 1, the average ID success rate is 62.1%62.1\% for CAST, 61.3%61.3\% with Centering alone, and 61.5%61.5\% with Gating alone, whereas combining both in full GSC jumps to 68.9%68.9\%. Rather than showing additive gains from two independently effective components, the results reveal a pronounced super-additive interaction effect. The authors attribute their complementarity to a conceptual division of labor—"Centering determines how process credit is assigned, whereas Gating determines when process supervision is applied"—but this remains a high-level intuition rather than a mechanistic explanation for why individual components underperform while their combination yields such a large jump.

Moreover, there are substantial disparities across individual games. Applying Centering alone improves success rates on Sokoban and Rush Hour, yet degrades Minesweeper performance from 44.7%44.7\% (CAST) to 34.8%34.8\%, before recovering to 50.9%50.9\% when combined with Gating. Similarly, the standalone effect of Gating is inconsistent across games. Relying on macro-averaged scores obscures these environment-specific dynamics. While the training curves on Minesweeper showing success-rate regression and response lengthening offer useful observations, they do not directly prove that the degradation is caused by process updates on all-success rollout groups, nor do they rule out confounding factors such as training stochasticity or shifts in the scale of process rewards.

I suggest evaluating all four factorial configurations (2×22 \times 2: CAST, Centering-only, Gating-only, and full GSC) within each game across multiple independent training seeds under an identical compute budget and unified checkpoint selection rule, explicitly reporting the interaction effect and its variance. A deeper mechanistic analysis should also track the proportion of all-success groups over training, the actual activation rate of the Gate, the relative magnitude of outcome versus process advantage terms, and the evolution of invalid/repetitive action rates and trajectory lengths. Finally, the authors should compare All-Success Gating against simpler baselines—such as uniformly scaling down or annealing the process supervision coefficient—to verify whether the gains genuinely stem from selectively gating off all-success groups rather than merely dampening the overall strength of process supervision.

Questions for authors

See Critical Weaknesses for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.