acceptodds
Under review as a conference paper at ICLR 2027

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

Abstract

Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose **CAST** (**C**redit **A**ssignment from **S**olver **T**eachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_pr5TOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CAST, a reinforcement learning framework that uses classical game solvers as turn-level teachers for LLM agents. CAST converts per-action changes in a solver’s state value (cost-to-go) into a shifted and shaped solver advantage that augments outcome-only RLVR signals, yielding dense, turn-level credit. The authors provide a theoretical interpretation showing that, under a soft-optimality assumption, maximizing solver advantage is equivalent to on-policy distillation from the solver without needing logits, and empirically demonstrate consistent gains across Sokoban, Minesweeper, and Rush Hour, with zero-shot transfer improvements on ALFWorld and WebShop.

Critical strengths

Technical novelty

  • The logit-free on-policy distillation interpretation is insightful: it connects solver-derived scalar signals to OPD via soft-optimal policies, bridging classic planning/solving and modern LLM distillation.

Experimental rigor and validation

  • Comprehensive in-domain and unseen-difficulty evaluations on three distinct games, with consistent improvements over multiple trained RLVR baselines. Zero-shot transfer results on ALFWorld and WebShop suggest the learned turn-level competencies generalize beyond the training domains.

Clarity of presentation

  • The method and signal construction are explained clearly with intuitive grounding in cost-to-go and shortest-path semantics. The theoretical statement (Theorem 2.1) is well-motivated and the assumptions are listed explicitly.
Critical weaknesses

Limited significance of the problem setting

  • The central motivation of the paper is long-horizon credit assignment under sparse terminal rewards. However, in the main experimental settings, CAST assumes access to a task-specific solver that can provide a reliable state value or cost-to-go estimate for arbitrary learner-visited states. Once such intermediate values are available, much of the difficult part of credit assignment---namely, determining which intermediate decisions are beneficial or harmful---has already been resolved by the solver. In this sense, CAST seems to primarily address how to integrate an existing value oracle into LLM RLVR, rather than how to solve credit assignment from sparse outcome supervision itself. Many realistic domains lack such oracles, and the paper’s learned-value ablation is limited to a single game that already can be solved by traditional DQN.

Technical limitations or concerns

  • In the practical method, the solver value is defined asVSolver(s)=−N(s),V^{\mathrm{Solver}}(s) = -N(s), where N(s)N(s) is the shortest-path cost-to-go. Therefore, the solver advantage used by CAST is constructed from a hard shortest-path value. However, in Appendix C.5, the derivation of ASolver(s,a)=τlog⁡πSolver(a∣s).A^{\mathrm{Solver}}(s,a)=\tau \log \pi_{\mathrm{Solver}}(a\mid s). relies on Assumption C.1, under which VSolver(s)=τlog⁡∑a′exp⁡(QSolver(s,a′)/τ),V^{\mathrm{Solver}}(s)=\tau \log\sum_{a'}\exp\left(Q^{\mathrm{Solver}}(s,a')/\tau\right), i.e., a soft log-sum-exp value. In general, this soft value is not identical to the hard shortest-path value used in the actual implementation. The wording in the main text that CAST is "exactly" or "mathematically equivalent" to on-policy distillation appears too strong.

  • For Minesweeper (partial observability), it is unclear whether the solver uses privileged access to the ground-truth board to compute N(s) or acts on information states only.

Experimental gaps or methodological issues

  • The paper motivates CAST partly by arguing that solver-trajectory SFT only covers expert-visited states, whereas CAST can provide supervision on student-visited states. This is a reasonable motivation, but the experiments do not include a solver-trajectory SFT baseline.

Missing related work or comparisons

  • The connection to potential-based reward shaping (Ng et al., 1999) is highly relevant: the per-step credit N(st)−N(st+1)N(s_t) − N(s_{t+1}) is a canonical potential difference. This connection should be made explicit and contrasted to clarify what is new.
Questions for authors

Scope of the credit-assignment claim.
CAST assumes a solver that provides reliable intermediate-state values. Could the authors clarify what part of credit assignment remains to be solved once such an oracle is available, and what application regime is intended when no reliable value oracle exists?

Hard value versus soft-value theory.
The implementation uses the hard cost-to-go value V(s)=−N(s)V(s)=-N(s), while the distillation derivation assumes a soft log-sum-exp value. Is the claimed equivalence to on-policy distillation exact, approximate under the small-signal assumption, or only valid up to a policy-gradient baseline argument?

Minesweeper under partial observability.
Does the Minesweeper teacher value depend only on the observable information state, or also on the fixed latent mine layout? If two latent boards induce the same information state, is the solver cost necessarily identical?

Direct solver-supervision baseline.
Do the authors have results against solver-trajectory SFT or a DAgger/AggreVaTe-style baseline using the same solver information? This would help isolate the benefit of CAST itself from the benefit of access to a strong oracle.

Relation to potential-based reward shaping.
How does CAST differ from using N(st)−N(st+1)N(s_t)-N(s_{t+1}) as a standard potential-based shaped reward? A direct comparison using the same solver signal would materially clarify the method's novelty.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

W1 / Q1. Solver signals, general-purpose LLM training, and non-game applications.

Our goal is to use solver signals to train a general-purpose LLM for reliable interaction. CAST turns value changes into credit for the learner's own sampled turns while retaining the terminal task objective. Solving credit assignment from sparse outcome supervision alone is not our contribution.

Games provide scalable instances, controllable difficulty, and reliable teacher signals for learning planning and feedback-driven decisions. Prior work also supports game-based transfer [1,2]. Our game-trained policies generalize zero-shot to ALFWorld and WebShop (Tables 2 and 13), where action selection does not require the source-game teacher.

Figure 4 already replaces the exact solver with a DQN value-network teacher, demonstrating that CAST's scalar interface accommodates learned evaluators. In other domains, an LLM teacher could serve as the solver, scoring task progress before and after learner actions. Differences between these scores would provide turn-level credit, enabling distillation without teacher logits. AgentPRM and VinePPO provide related approaches to process evaluation [3,4]. Such applications require an LLM teacher that can reliably assess task progress.

References

[1] LMGame-Bench: How Good are LLMs at Playing Games?

[2] SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning

[3] AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress

[4] VinePPO: Refining Credit Assignment in RL Training of LLMs

W2 / Q2. Hard values, soft values, and the theoretical interpretation.

For finite hard advantages, define qτ(a∣s)=exp⁡(Ahard(s,a)/τ)/Zτ(s)q_\tau(a\mid s)=\exp(A_{\rm hard}(s,a)/\tau)/Z_\tau(s). Then

=\tau\log q_\tau(a\mid s)+\tau\log Z_\tau(s).$$ The second term depends only on the state and cancels as a baseline in the population expected policy gradient with fixed scores, state distribution, and a given normalization constant. Therefore, linear hard advantages have a local log-preference equivalence. This preserves the interpretation of conveying action preferences without teacher logits, without requiring hard values to be equated with soft values. > **W2 / Q3. Minesweeper teacher under partial observability.** The Minesweeper teacher selects actions from revealed counts using constraint reasoning and mine probabilities. It derives its own flags rather than treating learner flags as facts. Its completion cost $K(s)$ counts the teacher's reveal actions plus one unflagging action for each unrevealed safe cell flagged by the learner; $K(s)=+\infty$ when the teacher cannot clear the board in its simulated continuation, for example when a probabilistic reveal hits a mine. **$K(s)$ is a teacher-rollout completion cost on the fixed latent board, not an information-state value averaged over compatible hidden layouts; identical observations can therefore yield different costs.** The environment maps infinite-cost boundaries to the finite raw scores below. The trainer then applies asinh compression and batch RMS normalization to these scores: | Transition | Raw solver score | | --- | --- | | Finite $\rightarrow$ finite | $K(s_t)-K(s_{t+1})$ | | Finite $\rightarrow$ infinite | $-K(s_t)$ | | Infinite $\rightarrow$ finite, nonterminal | $+K(s_{t+1})$ | | Infinite $\rightarrow$ infinite | $0$ | | Invalid action | $0$ | An undefined initial cost or an already-terminal state also receives zero. This produces finite scores without evaluating $\infty-\infty$. Finite-to-finite transitions account for over 80%, finite-to-infinite transitions for roughly 5%, infinite-to-finite transitions for less than 1%, and infinite-to-infinite transitions for roughly 10%.
AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3 / Q4. Solver-trajectory SFT and DAgger-style supervision.

We added solver-trajectory behavior cloning (BC) and learner-state DAgger, both starting from Qwen3-4B-Instruct-2507 and using the same Sokoban A⋆A^{\star} solver. BC learns from solver trajectories, while DAgger learns solver-labelled actions at learner-visited states. We evaluate action-only and templated-rationale targets, with one training run per method and target style.

The budget counts supervised action targets cumulatively across training, with a common target referenced to CAST's rollout environment-action count. We evaluate final training checkpoints.

BC and DAgger use the same Sokoban test tiers and Avg@4 protocol. DAPO and CAST scores retain the main experiment's run aggregation. On Unseen-Difficulty, the results are:

MethodTarget formatAvg@4 (%)
BCAction only22.1
BCTemplated rationale + action17.3
DAggerAction only19.5
DAggerTemplated rationale + action25.3
DAPOOutcome reward31.3
CASTOutcome + solver advantage34.8

CAST reaches 34.8% on Unseen-Difficulty, 9.5 percentage points above the strongest BC/DAgger variant. BC and DAgger can achieve higher ID performance than CAST, but this advantage does not extend to harder instances. Directly learning solver action labels yields limited generalization, whereas solver-guided turn-level credit generalizes better to unseen difficulty. The updated comparison is included in Table 10, Appendix E.1, with the budget definition.

Our best AggreVaTe result achieves 49.3% on Sokoban ID and 18.4% on Unseen-Difficulty, below CAST's reported scores on both settings.

W4 / Q5. Comparison with potential-based reward shaping.

On finite-cost transitions, CAST's cost difference is also a standard PBRS potential difference [1]. The distinction is how this signal enters the update.

With γ=1\gamma=1, complete rollouts from the same initial state within a prompt group, and zero potential at every terminal state, the accumulated PBRS shaping reward is a shared constant removed by group centering. Thus, aggregating PBRS rewards before GRPO group normalization does not change the trajectory advantages.

CAST instead adds the transformed and normalized immediate solver signal directly to turn-level advantages, allowing actions within the same trajectory to receive different credit.

References

[1] Policy Invariance under Reward Transformations: Theory and Application to Reward Shaping

Thank you for the detailed rebuttal and additional experiments. The responses have improved my assessment of the paper, particularly through the new experimental comparisons.

1. Scope and motivation. The clarification that CAST aims to transfer solver-derived feedback into general-purpose LLM training, rather than solving credit assignment from sparse rewards alone, is helpful. The DQN-based teacher experiment also supports the feasibility of using learned evaluators. However, the method still relies on reliable intermediate-state evaluations, which limits its applicability in domains without such evaluators.

2. Theoretical interpretation. The Gibbs distribution constructed from hard solver advantages provides a reasonable explanation of their implicit action preferences. However, this does not fully establish exact equivalence between the implemented CAST update and standard on-policy distillation, given the nonlinear transformation, batch normalization, and approximation assumptions. The theoretical claims should be qualified accordingly.

3. Minesweeper partial observability. The clarification that the teacher selects actions using observable information is useful. However, its completion cost is evaluated on the fixed latent board, meaning that the supervision can depend on hidden information even when the observable state is identical. This privileged supervision should be explicitly acknowledged as a limitation.

4. Solver imitation baselines. The additional BC and DAgger experiments substantially strengthen the paper. CAST achieves 34.8% on unseen difficulty, outperforming the strongest imitation baseline (25.3%). Although DAgger performs better in-domain, these results provide meaningful evidence that solver-guided turn-level RL generalizes better to unseen difficulty. This addresses one of my major experimental concerns.

5. Potential-based reward shaping. The distinction between trajectory-level reward shaping and directly incorporating solver signals into turn-level advantages is reasonable and clarifies the methodological contribution. Nevertheless, the relationship to potential-based advantage estimation deserves further discussion.

Overall assessment. The rebuttal addresses several important concerns, particularly the missing solver-imitation comparisons. My remaining reservations concern the theoretical equivalence claims and privileged supervision in Minesweeper, which have not been fully resolved.

Given these remaining concerns, I will maintain my original rating of 2 (Weak Reject).

Reviewer_VhKLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work proposes CAST, a new per-turn credit-assignment method to address the limitations of outcome-only rewards. The method uses either game solvers or a learned value network to assign per-turn credit, which is added to the outcome-based advantage. CAST performs better than baselines like GiGPO, GRPO, and DAPO across Sokoban, Minesweeper, and Rush Hour with Qwen3-4B-Instruct-2507, and the improvements transfer zero-shot to ALFWorld and WebShop. They also theoretically relate CAST to on-policy distillation (OPD).

Critical strengths
  1. Per-turn credit is an important problem with many practical applications, like improving training convergence, reducing redundant turns, etc.
  2. Results are compared against a wide variety of methods, such as GiGPO, GRPO, DAPO and GSPO.
  3. Training and evaluation across multiple games (Sokoban, Minesweeper, and Rush Hour) strengthen the CAST methodology.
Critical weaknesses
  1. All the training was done with only one model, Qwen3-4B-Instruct-2507. Adding a model of a different size, like Qwen3-8B or 32B, would strengthen the paper or at least provide results with a thinking variant (Qwen3-4B-thinking-2507/ Qwen3-4B think vs nothink).
  2. Currently, it is difficult to understand what models learned about the game and what is causing generalization to ALFWorld and WebShop. Since all the training was on the Instruct variant, the models could have just deliberated more (generating more tokens), similar to the thinking variant, which could have resulted in improved OOD performance.
  3. The proposed method relies on a game solver or learned value function; there is still a gap in how we could use it for non-game scenarios.
  4. The CAST and OPD theoretical connection is not fully justified. First-order approximation of the asinh function is used in Lemma C.6, but in Theorem 2.1, exact equivalence is claimed, ignoring the approximation error. Also, the proof assumes a soft-optimal solver, but it is unclear whether this assumption holds for the empirical experiments demonstrated here.
Questions for authors
  1. Have the authors tried different ways of combining the solver feedback with the advantage term, apart from adding alpha*solver score? Intuitively, I would prefer to penalize the redundant turn even if the outcome reward was positive. Currently, a wrong turn could still receive a positive advantage if the overall trajectory has a positive advantage. It would be a good ablation to check different forms of combining the solver feedback.
  2. Training with the thinking variant will help find the cause of generalization to ALFWorld and Webshop. Whether training just made the Instruct variant think more or some other form of capability boost.
  3. Could the authors provide the tokens generated by models before and after training for all the evaluations?
  4. The authors show that a learned value function can replace the solver in Rush Hour. Could they discuss the additional compute required to train this value function and its trade-offs against outcome-only rewards? It would be great if the authors could extend it to general tasks. i.e Value function training + CAST vs Outcome only reward compute.
  5. Adding one task where outcome-only reward results in entropy collapse where CAST helps, would improve the paper. For example, tools use heavy tasks; outcome-only reward might reinforce redundant tool calls, which results in entropy collapse, whereas per-turn credit avoids it. This is one of the reasons why I believe in penalizing the turns instead of reducing the reward (1st question). Also, demonstrating this solves Weakness 3.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

W1. Different model sizes and thinking-enabled policies.

We added Qwen3-1.7B and Qwen3-8B experiments with thinking both disabled and enabled. With thinking disabled, Table 11, Appendix E.2 reports the following Sokoban ID Avg@4 percentages, averaged over three runs:

ModelGRPOGSPODAPOGiGPOCAST
Qwen3-1.7B17.519.022.319.226.4
Qwen3-8B35.139.254.137.766.8

With thinking enabled, CAST and DAPO share the model, training data, schedule, and configured rollout budget within each size, using 24,576 response tokens per episode. Each cell reports one training run's Sokoban ID Avg@4:

ModelDAPOCASTGain (percentage points)
Qwen3-1.7B41.950.0+8.1
Qwen3-8B72.484.9+12.5

CAST remains effective with thinking both disabled and enabled, supporting its applicability across different base models. The thinking-enabled results are included in Table 12, Appendix E.3. The same trained policies are used for the transfer experiments below.

W2 / Q2. Transfer from game training, thinking variants, and generation length.

We evaluated the Sokoban-trained thinking policies from Table 12 zero-shot on ALFWorld and WebShop, with no target-domain fine-tuning and the same 24,576-token response budget:

ModelMethodALFWorld seen (%)WebShop (%)
1.7BReAct/Base9.84.5
1.7BDAPO23.64.1
1.7BCAST25.47.4
8BReAct/Base63.825.8
8BDAPO68.431.1
8BCAST68.832.3

CAST improves transfer success in all four comparisons while generating fewer tokens than DAPO (Q3), even with thinking already enabled. These results argue against increased generation alone explaining the transfer gains. The success rates are included in Table 13, Appendix E.3.

A plausible transferable capability is goal-directed interaction: using feedback to select actions, check constraints, and recover from mistakes. Solver-guided credit links sampled actions to goal progress during training.

W3 / Q5. Non-game applications and outcome-only entropy collapse.

Games as training grounds. Our goal is to train general-purpose LLMs for reliable interaction, including planning under constraints, acting on observations, and recovering through feedback. Games provide scalable instances, controllable difficulty, and inexpensive, verifiable teacher signals. Game-based transfer to planning, reasoning, and agent tasks is also supported by prior work [1,2]. Our Tables 2 and 13 demonstrate zero-shot use on ALFWorld and WebShop without a target-domain teacher.

Non-game applications. In other domains, an LLM teacher could serve as the solver, scoring task progress before and after learner actions. Differences between these scores would provide turn-level credit, enabling distillation without teacher logits. AgentPRM and VinePPO offer related approaches to learned and rollout-estimated process evaluation [3,4], while our Figure 4 already demonstrates a learned teacher. Direct non-game training requires a reliable evaluator and remains an extension of the present experiments.

Entropy and performance. CAST does not directly optimize entropy; it uses credit assignment to help the model distinguish beneficial from harmful actions. Higher entropy alone does not imply better task performance and can mask input-agnostic templates [5]. Regarding your preference for penalizing redundant turns, neither CAST-Pen nor CAST-Mul improved success in our Q1 experiments, supporting the original credit combination.

References

[1] LMGame-Bench: How Good are LLMs at Playing Games?

[2] SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning

[3] AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress

[4] VinePPO: Refining Credit Assignment in RL Training of LLMs

[5] RAGEN-2: Reasoning Collapse in Agentic RL

W4. Lemma C.6 and Theorem 2.1.

CAST training uses the full asinh transformation because it is strictly monotonic and preserves the ordering of the raw scores. Our Taylor expansion is used only to characterize the local connection to distillation. Note that we restrict the theorem to an unclipped, population-level local result with fixed scores, state distribution, and normalization constant, and retain the first-order expansion remainder bound ∣r(A)∣≤A2/(33)|r(A)|\le A^2/(3\sqrt3), where AA is the raw score minus 1. Actual training requires neither teacher logits nor a teacher satisfying the soft-optimal assumption.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q1. Alternative combinations and penalties for redundant turns.

We tested CAST-Pen and CAST-Mul, which penalize selected non-progress turns even with positive outcome advantage. Let AA be the outcome advantage, ata_t the raw solver score, and Pt=1P_t=1 a checked non-progress turn. Both use gt=asinh⁡(at−Pt)g_t=\operatorname{asinh}(a_t-P_t) and ht=gt/(RMS⁡batch(g)+ϵ)h_t=g_t/(\operatorname{RMS}_{\mathrm{batch}}(g)+\epsilon), with α=0.1\alpha=0.1 and ϵ=10−8\epsilon=10^{-8}.

A^tPen={min⁡(A,0)+αht,Pt=1,A+αht,Pt=0.\widehat A_t^{\mathrm{Pen}}=\begin{cases}\min(A,0)+\alpha h_t,&P_t=1,\\ A+\alpha h_t,&P_t=0.\end{cases}

A^tMul={−α∣A∣∣ht∣,Pt=1,A[1+αsign⁡(A)tanh⁡(ht)],Pt=0.\widehat A_t^{\mathrm{Mul}}=\begin{cases}-\alpha|A||h_t|,&P_t=1,\\ A\left[1+\alpha\operatorname{sign}(A)\tanh(h_t)\right],&P_t=0.\end{cases}

MethodSokoban ID Avg@4 (%)Difference from CAST (percentage points)
CAST77.0—
CAST-Pen72.1−4.9
CAST-Mul63.0−14.0

Both variants assign negative advantage to the checked non-progress turns with positive outcome advantage, but neither improves success over the reported CAST result. These observations favor retaining the original additive combination.

Q3. Generated tokens before and after training.

The tables report mean generated tokens per episode over all attempts, including reasoning and actions but excluding repeated history, observations, and chat-template tokens. For the original game evaluations, ReAct/Base is the policy before training and the other rows are the trained policies from Table 1:

MethodSokoban IDSokoban UnseenMinesweeper IDMinesweeper UnseenRush Hour IDRush Hour Unseen
ReAct/Base8,87911,49010,40310,77211,41711,482
GRPO6,57411,2029,1009,4697,3499,466
GSPO8,28813,02310,62711,09712,91913,652
DAPO7,00111,6068,7489,37012,43513,745
GiGPO6,96211,5107,9548,1559,15910,766
CAST7,43211,72110,63611,7379,88411,733

CAST generates more than DAPO on Sokoban and Minesweeper but less on Rush Hour. Compared with the untrained ReAct/Base policy, CAST generates less on Sokoban ID and Rush Hour ID and more in the other four settings, while achieving higher success than both policies in all six settings. Thus, the gains have no uniform length-increase pattern.

For the thinking-enabled Sokoban experiments in Table 12, we additionally retokenized the saved validation responses using the same generated-text counting convention. The columns compare evaluations before and after training under the 24,576-token response budget.

ModelDAPO beforeDAPO afterCAST beforeCAST after
Qwen3-1.7B9,56817,1679,51714,408
Qwen3-8B9,32315,5769,32912,294

Training increases generation for both methods, but CAST uses 16.1% fewer tokens at 1.7B and 21.1% fewer at 8B than DAPO while achieving higher success. CAST also has a higher average interaction-turn count in both comparisons, indicating more compact per-turn generation rather than simply fewer environment steps.

For the newly added thinking-enabled transfer evaluation, the following table includes the frozen base policy and both trained policies. Counts sum the new completion-token IDs across the episode without recounting history:

ModelMethodALFWorld seen tokens/episodeWebShop tokens/episode
1.7BReAct/Base4,5015,452
1.7BDAPO6,2227,606
1.7BCAST4,7235,504
8BReAct/Base2,9603,978
8BDAPO3,7315,622
8BCAST3,0015,199

On ALFWorld/WebShop, CAST reduces generation relative to DAPO by 24.1%/27.6% at 1.7B and 19.6%/7.5% at 8B, with higher success in all four comparisons. Higher success with less generation supports more economical use of reasoning for interaction; increased length alone does not explain CAST's gains. These results do not imply that shorter outputs are always better.

Q4. Learned Rush Hour teacher: training cost and online overhead.

The learned value teacher has fewer than one million parameters. Its complete one-time preparation costs about 2.2 GPU·h, including training, evaluation, and checkpointing; an online scalar query takes approximately 1.6 ms per state on CPU, or 21.3 µs with a cache hit. The teacher can be reused across policy-training runs, amortizing its preparation cost. Together with the learned-teacher results in Figure 4, these measurements show that useful process supervision can be provided with modest one-time preparation and millisecond-scale online overhead.

I appreciate the authors for the new results with the thinking variant and the token budget table. Most of my concerns are addressed, so I'm increasing my score to 3. My remaining concern is that there is no example of training with CAST outside games.

AuthorsOpenReview ↗

Follow-up Response

Dear Reviewer VhKL,

Thank you for taking the time to review the new results for the thinking variant and the token budget table. We appreciate your constructive feedback and are glad that the additional results addressed most of your concerns. To address your remaining concern, we will include an experiment in the final version in which we train with CAST in a non-game environment and evaluate its effectiveness in that setting.

Best regards,

Authors of Submission 12135

Reviewer_eKGWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CAST augments outcome-based reinforcement learning for LLM agents with turn-level supervision derived from changes in a game solver’s cost-to-go. The signal is compressed using asinh, normalized by batch RMS, and added to the outcome advantage. Experiments cover three games, harder held-out difficulty levels, and zero-shot transfer to ALFWorld and WebShop. The paper also proposes an interpretation as logit-free on-policy distillation and evaluates a learned value network as an alternative teacher.

Critical strengths
  1. Consistent empirical improvements. CAST outperforms all trained baselines across all six game/settings combinations. Against its DAPO backbone, average success increases from 44.7% to 62.1% in-domain and from 18.7% to 28.4% on unseen difficulty. Three independent runs and variability bands strengthen this evidence.

  2. Practical supervision with encouraging transfer. Scalar solver feedback provides a simple way to supervise learner-visited states without teacher token distributions. The zero-shot transfer results and Rush Hour experiment using a learned value network suggest utility beyond memorizing exact solver trajectories.

Critical weaknesses
  1. The claimed theoretical equivalence is not established as stated. Lemma C.6 retains an action-dependent O(A2)O(A^2) Taylor remainder, but Theorem 2.1 drops it and claims exact gradient equality. Moreover, ordinary shortest-path optimality does not establish the soft-optimal identity assumed in C.1; using V=−NV=-N generally differs from the required log-sum-exp value. Finally, Corollary C.8 optimizes a KL-regularized objective, whereas CAST’s proposed objective uses cross-entropy. Remark C.9 acknowledges this distinction, so the claimed pure-distillation limit and teacher-matching interpretation do not follow for CAST itself. These issues affect a central contribution rather than a peripheral justification.

  2. The comparisons do not isolate the benefit of CAST over alternative uses of the same oracle. The trained baselines lack solver supervision, while no solver-trajectory SFT or learner-state imitation/cost-to-go baseline is evaluated. This is especially important given the connection to AggreVaTe, which already uses expert cost information in interactive imitation learning. The current results demonstrate the value of adding solver information, but do not establish the advantage of the proposed way of using it. Ablations of asinh and normalization do not resolve this attribution issue.

Questions for authors
  1. Theory: Can Theorem 2.1 be reformulated as an approximate, frozen-visitation surrogate result with an explicit remainder bound? Please distinguish the implemented hard-solver values from soft values, and clarify which conclusions actually apply to CAST rather than the separate KL-regularized objective.

  2. Comparisons: How does CAST compare with solver-trajectory SFT and learner-state solver imitation under matched oracle-query and training budgets?

  3. Minesweeper implementation: Appendix C.3 defines K(s)=+∞K(s)=+\infty when the solver rollout fails. How are finite-to-infinite, infinite-to-finite, and infinite-to-infinite differences handled? Since K(s)K(s) depends on the fixed hidden layout, please clarify whether this is a sampled teacher return or an information-state value, and report how frequently these cases occur.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

W1 / Q1. Lemma C.6, Theorem 2.1, and the distillation interpretation.

The connection between scalar scores and distillation can be retained and explicitly stated as a local first-order relationship. Specifically, for fixed, finite scores x(s,a)x(s,a) on a finite action set, let A=x−1A=x-1 and qτ(a∣s)∝eA(s,a)/τq_\tau(a\mid s)\propto e^{A(s,a)/\tau}, where τ>0\tau>0. Fix the state distribution μ\mu and a given normalization constant D>0D>0, and define the unclipped local objective Lsol=(α/D)Eμ,πθ[asinh⁡(x)]L_{\rm sol}=(\alpha/D)\mathbb E_{\mu,\pi_\theta}[\operatorname{asinh}(x)]. Then

=-\beta\nabla_\theta\mathbb E_\mu H(\pi_\theta,q_\tau)+e, \qquad \|e\|\le \frac{\alpha}{3\sqrt3D} \mathbb E_{\mu,\pi_\theta}\!\left[\|\nabla_\theta\log\pi_\theta(a\mid s)\|A^2\right], \quad \beta=\frac{\alpha\tau}{\sqrt2D}.$$ The closed-form solution in C.8 belongs to a separate KL objective. In the final version, we will improve the wording of this distribution-matching conclusion for CAST while retaining the interpretation that the cross-entropy objective favors high-scoring actions. Note that actual training uses the full asinh transformation; scores are not computed using a Taylor approximation. > **W2 / Q2. Solver-trajectory SFT and learner-state solver imitation.** We added solver-trajectory behavior cloning (BC) and learner-state DAgger, both starting from Qwen3-4B-Instruct-2507 and using the same Sokoban $A^{\star}$ solver. BC learns from solver trajectories, while DAgger learns solver-labelled actions at learner-visited states. We evaluate action-only and templated-rationale targets, with one training run per method and target style. The budget counts supervised action targets cumulatively across training, with a common target referenced to CAST's rollout environment-action count. We evaluate final training checkpoints. BC and DAgger use the same Sokoban test tiers and Avg@4 protocol. DAPO and CAST scores retain the main experiment's run aggregation. On Unseen-Difficulty, the results are: | Method | Target format | Avg@4 (%) | | --- | --- | --- | | BC | Action only | 22.1 | | BC | Templated rationale + action | 17.3 | | DAgger | Action only | 19.5 | | DAgger | Templated rationale + action | 25.3 | | DAPO | Outcome reward | 31.3 | | CAST | Outcome + solver advantage | **34.8** | **CAST reaches 34.8% on Unseen-Difficulty, 9.5 percentage points above the strongest BC/DAgger variant.** BC and DAgger can achieve higher ID performance than CAST, but this advantage does not extend to harder instances. Directly learning solver action labels yields limited generalization, whereas solver-guided turn-level credit generalizes better to unseen difficulty. The updated comparison is included in **Table 10, Appendix E.1**, with the budget definition. Our best AggreVaTe result achieves **49.3% on Sokoban ID** and **18.4% on Unseen-Difficulty**, below CAST's reported scores on both settings. > **Q3. Minesweeper completion cost under partial observability.** The Minesweeper teacher selects actions from revealed counts using constraint reasoning and mine probabilities. It derives its own flags rather than treating learner flags as facts. Its completion cost $K(s)$ counts the teacher's reveal actions plus one unflagging action for each unrevealed safe cell flagged by the learner; $K(s)=+\infty$ when the teacher cannot clear the board in its simulated continuation, for example when a probabilistic reveal hits a mine. **$K(s)$ is a teacher-rollout completion cost on the fixed latent board, not an information-state value averaged over compatible hidden layouts; identical observations can therefore yield different costs.** The environment maps infinite-cost boundaries to the finite raw scores below. The trainer then applies asinh compression and batch RMS normalization to these scores: | Transition | Raw solver score | | --- | --- | | Finite $\rightarrow$ finite | $K(s_t)-K(s_{t+1})$ | | Finite $\rightarrow$ infinite | $-K(s_t)$ | | Infinite $\rightarrow$ finite, nonterminal | $+K(s_{t+1})$ | | Infinite $\rightarrow$ infinite | $0$ | | Invalid action | $0$ | An undefined initial cost or an already-terminal state also receives zero. This produces finite scores without evaluating $\infty-\infty$. Finite-to-finite transitions account for over 80%, finite-to-infinite transitions for roughly 5%, infinite-to-finite transitions for less than 1%, and infinite-to-infinite transitions for roughly 10%.
AuthorsOpenReview ↗

Anticipating Your Response

Dear Reviewer eKGW,

Thank you again for your thoughtful feedback. As the discussion period draws to a close, we wanted to follow up on our response, particularly our clarification of the scope of the theoretical result and the new comparisons with BC and DAgger.

If you feel that our clarifications and additional results address your concerns, we would appreciate it if you could revisit your overall assessment. We would also be happy to discuss any remaining questions or concerns.

Best regards,

Authors of Submission 12135

AuthorsOpenReview ↗

Anticipating Your Response

Dear Reviewer eKGW,

As the discussion period draws to a close, we wanted to let you know that we have uploaded the revised PDF, with the changes highlighted in blue.

Theorem 2.1 now states a local, first-order result for the unclipped solver term, and Appendix C.5 gives an explicit bound on the approximation error. Table 10 compares CAST with BC and DAgger using the same Sokoban solver. Appendix C.3 clarifies how we assign finite scores when the Minesweeper teacher cannot complete a rollout.

We would be grateful if you could take these revisions into account in your final assessment. If any concerns remain, we would be happy to discuss them. Thank you again for your time and careful review.

Best regards,

Authors of Submission 12135

Reviewer_sFqyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAST adds solver feedback to reinforcement learning for multi-turn LLM agents with sparse terminal rewards. It scores each learner action by the change in estimated remaining work, applies asinh compression and batch RMS normalization then adds the result to DAPO's outcome advantage. The evaluation covers three games, harder within-game tiers, zero-shot transfer to ALFWorld and WebShop and a learned Rush Hour value teacher. The empirical gains are useful, but the exact theoretical claim and the teacher specification need correction before acceptance.

Critical strengths
  1. Useful gains under a controlled RL backbone. The comparison to DAPO holds the base model, training data and rollout budget fixed and adds the solver signal. Table 1 reports gains of about 17 percentage points in the in-domain average and 10 points on unseen difficulty, with improvements in every game and tier. Figure 3 includes standard-deviation bands over three runs and shows that the benefits repeat during training. This is useful evidence that solver feedback helps multi-turn RL in this setting even if the theoretical account is narrowed.

  2. Practical supervision with initial transfer evidence. The scalar interface allows a game solver to supervise sampled actions without supplying token distributions. In Table 2 the separate game-trained policies obtain the highest source-policy averages on both ALFWorld and WebShop, with about 30% overall success compared with about 25% for the frozen base. The learned Rush Hour value teacher in Figure 4 retains much of the exact teacher's benefit without solver-distance targets. Figure 6 also provides a linear-score control using the same solver with RMS normalization on Sokoban, supporting the score stabilization choices. Together these checks support a useful training recipe, with applicability currently demonstrated through separate source-game policies, two transfer domains and one learned-teacher game.

Critical weaknesses
  1. The exact and global claims are not established. Lemma C.6 retains an action-dependent nonlinear remainder in Equation 16, but Step 3 of the Theorem 2.1 proof drops it before asserting exact equality. Small advantages can justify an approximation, but do not make that remainder vanish from the gradient. The hard and proxy costs used in the implementation are also not shown to satisfy the soft-value relation the theorem assumes. Appendix C.5.6 then treats the local update under fixed state visitation as the gradient of a global augmented return without accounting for how actions change later scores. Its task-error bound also lacks conditions on student optimization and state coverage. A correctly scoped local or approximate account could still explain the recipe, but the current argument does not establish a headline contribution. The authors should correct or narrow the theoretical claims and clearly explain which conclusions apply to the implemented method.

  2. Failed teacher rollouts and the common protocol need specification. Appendix C.3 assigns infinite Minesweeper completion cost when the peek-free solver fails, including a losing probabilistic reveal. The documented cost difference and dead-state penalty in Section 2.2 do not specify a finite score for every combination of infinite starting or ending costs. The authors should document the finite rules actually used, their observed frequencies and how these rollout labels relate to the information-state values described in the main text. The illegal-action and token-budget descriptions also conflict between sections of the paper. Three-run curves support the main effect, but Tables 1 and 2 lack uncertainty and Figure 6 does not specify run aggregation or uncertainty. These gaps make it difficult to audit the teacher signal and common comparison protocol, which need to be resolved before the empirical contribution can support acceptance.

I would raise the rating to 3 if the authors correct or withdraw the unsupported theoretical claims, document the finite teacher rules and common protocol used for the reported runs and provide existing-run evidence that preserves the substantial empirical gains.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

W1. Lemma C.6 and Theorem 2.1.

CAST directly uses solver scores and the full asinh transformation to provide turn-level supervision; the computation does not require the teacher to be soft-optimal. We restrict the theorem to an unclipped, population-level local result with fixed scores, state distribution, and normalization constant, and will add remainder bounds in the final version. The method's practical value remains supported by controlled experiments: in Table 1, on the same DAPO backbone, the average ID/Unseen success rates increase from 44.7%/18.7% to 62.1%/28.4%, respectively.

W2. Minesweeper teacher cost and finite-score rules.

The Minesweeper teacher selects actions from revealed counts using constraint reasoning and mine probabilities. It derives its own flags rather than treating learner flags as facts. Its completion cost K(s)K(s) counts the teacher's reveal actions plus one unflagging action for each unrevealed safe cell flagged by the learner; K(s)=+∞K(s)=+\infty when the teacher cannot clear the board in its simulated continuation, for example when a probabilistic reveal hits a mine. K(s)K(s) is a teacher-rollout completion cost on the fixed latent board, not an information-state value averaged over compatible hidden layouts; identical observations can therefore yield different costs.

The environment maps infinite-cost boundaries to the finite raw scores below. The trainer then applies asinh compression and batch RMS normalization to these scores:

TransitionRaw solver score
Finite →\rightarrow finiteK(st)−K(st+1)K(s_t)-K(s_{t+1})
Finite →\rightarrow infinite−K(st)-K(s_t)
Infinite →\rightarrow finite, nonterminal+K(st+1)+K(s_{t+1})
Infinite →\rightarrow infinite00
Invalid action00

An undefined initial cost or an already-terminal state also receives zero. This produces finite scores without evaluating ∞−∞\infty-\infty.

Finite-to-finite transitions account for over 80%, finite-to-infinite transitions for roughly 5%, infinite-to-finite transitions for less than 1%, and infinite-to-infinite transitions for roughly 10%.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W2. Evaluation protocol, uncertainty in Tables 1 and 2, and Figure 6.

We corrected the episode-termination description in Section 3.1 (line 315), replacing "an illegal move" with "a game-specific failure", consistent with the invalid-action handling already described in Appendix C.1. We also corrected Appendix B.2 (lines 854–855), replacing the per-turn token limit with a cumulative response budget of 16,384 tokens per episode, consistent with Appendix C.6.2. These two corrections are highlighted in blue.

We recovered the fixed-step scores for all three runs of every trained method in Table 1. Below, values are success percentages, reported as mean ± standard deviation across the three runs, using the original aggregation convention (population standard deviation).

MethodSokoban IDSokoban UnseenMinesweeper IDMinesweeper UnseenRush Hour IDRush Hour Unseen
GRPO71.3 ± 3.530.5 ± 3.09.7 ± 0.80.5 ± 0.253.8 ± 3.926.7 ± 4.2
GSPO72.6 ± 1.428.3 ± 2.228.2 ± 13.85.0 ± 3.424.8 ± 17.211.5 ± 8.7
DAPO71.9 ± 2.431.3 ± 1.729.8 ± 5.67.3 ± 1.632.5 ± 23.017.4 ± 12.3
GiGPO70.6 ± 2.929.6 ± 4.113.7 ± 7.41.4 ± 1.351.9 ± 8.631.3 ± 6.8
CAST77.0 ± 2.634.8 ± 0.744.7 ± 10.811.0 ± 5.864.7 ± 3.839.5 ± 6.6

CAST improves over DAPO in each of the six game/difficulty settings. The ID gains are 5.1, 14.9, and 32.2 percentage points for Sokoban, Minesweeper, and Rush Hour, respectively.

For Table 2, each policy is trained on the source game named in the column and evaluated zero-shot on the target domain. Rows compare methods, while columns distinguish the three source-game policies. Values are Avg@4 success percentages, reported as mean ± standard deviation over three runs.

ALFWorld seen

MethodSokoban-trainedMinesweeper-trainedRush Hour-trained
GRPO31.5 ± 3.022.3 ± 5.427.9 ± 5.8
GSPO34.2 ± 1.622.0 ± 15.540.0 ± 9.6
DAPO37.3 ± 2.535.2 ± 3.318.6 ± 13.3
GiGPO35.6 ± 2.624.9 ± 2.732.1 ± 5.5
CAST38.4 ± 2.033.9 ± 3.341.4 ± 4.2

WebShop

MethodSokoban-trainedMinesweeper-trainedRush Hour-trained
GRPO20.4 ± 0.714.0 ± 2.618.0 ± 2.0
GSPO18.0 ± 1.313.1 ± 9.720.9 ± 2.4
DAPO17.5 ± 1.118.8 ± 1.013.6 ± 9.7
GiGPO17.2 ± 1.920.2 ± 5.516.4 ± 1.6
CAST23.2 ± 4.225.1 ± 2.619.7 ± 0.9

The "Average" columns in Table 2 average the three source-game policies within each target domain; "Overall" averages the two domain averages. CAST obtains 37.9% on ALFWorld, 22.7% on WebShop, and 30.3% Overall. The uncertainty values above are supplied here in addition to the means currently displayed in Tables 1 and 2.

Figure 6 reports one run per setting, with no aggregation across runs or cross-run uncertainty bands. It separates two design questions: the left column varies the solver-advantage weight α\alpha, while the right column removes or replaces asinh compression and batch RMS normalization using the same solver. The top row reports validation Avg@4 and the bottom row reports training reward. Curves use EMA smoothing with coefficient 0.6.

Reviewer_sFqyOpenReview ↗

Thank you

Thank you for the detailed response. The finite teacher rules, protocol corrections and three-run results address my empirical reporting concerns. The gains over DAPO is a reasonable improvment and I appreciate the effort to recover and explain these results. Narrowing the theory to a local result is a helpful response to my main concern. With that correction I am happy to support weak acceptance.

AuthorsOpenReview ↗

Follow-up Response

Dear Reviewer sFqy,

Thank you for taking the time to read our response and follow up. Your feedback helped us clarify the scope of our theoretical result and report the experiments more transparently. We appreciate your support and will incorporate these changes into the final version.

Best regards,

Authors of Submission 12135

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.