acceptodds
Under review as a conference paper at ICLR 2027

GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents

Abstract

Recently, Group Relative Policy Optimization (GRPO) has been widely adopted to train vision-language model (VLM) agents for diverse tasks. However, the context lengths of trajectory samples are too long for training VLM agents with GRPO to execute open-world tasks, which usually consumes too many training resources to remain practical. To address this issue, we propose GROW, a reinforcement learning (RL) framework for open-world VLM agents that decomposes trajectory samples into state-action samples, whose context lengths are shorter than those of trajectory samples, thereby reducing the required training resources. After decomposition, GROW performs credit assignment with temporal discounting on state-action samples, thereby converting sparse reward signals into dense ones to guide efficient learning, and it then computes advantages among the state-action samples within the same rollout groups. We further provide a surrogate analysis indicating that GROW retains the training signals for the whole trajectory samples in the optimization objective that can reinforce the successful trajectory samples and suppress the failed ones, while avoiding direct training on trajectory samples. Experiments on more than 800 Minecraft tasks show that our method can achieve higher average success rates and fewer steps than recent state-of-the-art approaches.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Open-world Agent, VLM, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_sqVVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GROW, which aims to address the challenges—namely, excessive context length and high computational resource consumption when training open-world Vision-Language Model (VLM) agents using Group Relative Policy Optimization (GRPO) on long interaction trajectories. GROW decomposes full trajectories into individual state-action samples. It propagates sparse, trajectory-level rewards to these individual samples via temporal discounting and then calculates relative advantages within each rollout group to optimize the policy. The authors provide a surrogate analysis, arguing that under certain assumptions, this approach preserves the optimization signals present at the trajectory level while avoiding the need to train directly on full trajectories. Experiments were primarily conducted on a Minecraft benchmark comprising over 800 tasks, spanning embodied manipulation, GUI interaction, and combat. The results show that GROW achieves success rates of 59.6%, 68.4%, and 49.0% across these three categories, respectively, demonstrating superior overall performance compared to trajectory-level GRPO and PPO. The authors also conducted extended experiments in simulated environments, including Language Table and WebShop. Overall, the paper addresses a problem of practical significance, proposes a relatively straightforward method, and reports positive experimental results. However, the degree of independent innovation in the core algorithm—relative to existing policy optimization methods—as well as the strength of the theoretical analysis in explaining the actual optimization objectives, still need further validation.

Critical strengths
  1. It offers a simple yet promising solution to a practical bottleneck in training long-horizon VLM agents. Open-world tasks typically require multiple rounds of visual observation and action interaction; the cumulative context of a full trajectory imposes significant demands on memory and computational resources. By decomposing training data into individual state-action samples, GROW drastically reduces the context length per training sample. The paper reports that the context length for GROW’s state-action samples is under 1K tokens, whereas trajectory-level GRPO can reach up to 32K tokens. This design offers clear engineering value and provides a practical pathway for reinforcement learning in long-horizon agent training.
  2. The experimental scope is extensive and demonstrates compelling performance improvements. The authors evaluated GROW across more than 800 Minecraft tasks, compared it against trajectory-level GRPO, PPO, and initialization policies, and conducted evaluations on both in-distribution (ID) and out-of-distribution (OOD) generalization. GROW achieved high success rates across embodied, GUI, and combat tasks; notably, in GUI tasks, the success rate increased from 43.9% (with PPO) to 68.4%. Furthermore, the authors conducted additional experiments on Language Table and WebShop, providing preliminary evidence that the method may be applicable to various interactive environments.
Critical weaknesses
  1. The core algorithmic innovation lacks sufficient novelty, and the fundamental advantages of GROW over existing policy optimization methods have not been clearly established. The primary design of GROW involves distributing trajectory-level rewards to individual state-action samples via temporal discounting, followed by group-wise reward normalization and advantage calculation. Structurally, this approach is closely related to existing methods based on reward shaping, temporal discounting, and local policy gradient optimization. Although the paper introduces a "surrogate analysis," it remains unclear whether this analysis reveals a mechanism that is substantively distinct from conventional reward propagation and policy optimization techniques. Specifically, GROW calculates advantages at the level of individual state-action samples, whereas standard GRPO typically operates at the level of complete responses or trajectories. This distinction may simultaneously alter reward normalization, sample weighting, and gradient estimation. Consequently, the current results do not sufficiently clarify whether the performance gains stem from the contextual advantages provided by trajectory decomposition, or from the specific modifications to reward propagation and sample weighting. It is recommended to include more granular ablation studies—for instance, by isolating the effects of temporal discounting, varying the reward normalization method, and comparing different optimization strategies while keeping the computational budget constant.
  2. The theoretical analysis relies on strong, simplifying assumptions, which are insufficient to fully substantiate the authors' claims regarding the preservation of optimization signals at the trajectory level. The authors' surrogate analysis assumes that trajectory lengths within the same rollout group are approximately equal; based on this, they decompose the GROW optimization objective into trajectory-level signals and terms related to temporal discounting. However, in open-world tasks, trajectory lengths can vary significantly—with successful and failed trajectories often exhibiting different length distributions. In such scenarios, the mean and variance calculated across individual state-action samples may be skewed by trajectory lengths, thereby altering the relative weighting of different trajectories and their constituent actions. Furthermore, while GROW computes policy probability ratios for each state-action sample independently, trajectory-level policy updates involve the joint probability of the entire trajectory. The authors need to clarify the conditions under which their current objective effectively approximates the optimization direction of trajectory-level GRPO, and quantify the potential bias introduced by variations in trajectory length and the use of action-wise probability ratios. The current analysis serves more as an interpretive decomposition of the objective function rather than a rigorous proof that GROW preserves the optimization properties of trajectory-level GRPO in the general case.
Questions for authors

The core claim of the paper is that GROW retains the essential optimization signals of trajectory-level GRPO while avoiding the need to train directly on full trajectories. Could the authors clarify: when trajectory lengths within a rollout group vary significantly, does the uniform calculation of reward means, standard deviations, and relative advantages across all state-action samples lead to disproportionately high optimization weights for longer trajectories—thereby shifting the optimization direction away from that of the original trajectory-level GRPO?

6 comments

1. Response to Weakness 1: Insufficient Algorithmic Novelty and Lack of Comprehensive Ablation Studies

We thank the reviewer for these comments. We clarify GROW's novelty and address the suggested ablations below.

1.1 GROW's Novelty Lies in Effectively Optimizing Trajectory Samples via State-Action Decomposition for Open-World Tasks

Although methods like temporal discounting have been used in prior work, GROW's novelty lies in decomposing trajectories into state-action samples and allocating trajectory-level rewards to these samples through temporal discounting. This enables GROW to optimize state-action samples from complete rollouts and train on open-world tasks whose full trajectory samples would otherwise exceed the context limit and thus impose substantial demands on training resources.

1.2 Ablations of Temporal Discounting, Reward Normalization, and Optimization Strategies Validate GROW's Effectiveness

The reviewer suggests three types of ablations. For temporal discounting, Table 4 in the main paper already reports results with γ = 1.0, which removes discounting. Since both settings use the same number of training steps and have similar GPU-hours per training step, their total GPU-hours are approximately equal.

For optimization strategies, Table 3 evaluates PPO applied to state-action samples. Each PPO training step takes longer than a GROW training step, resulting in higher GPU-hours under the same number of training steps, yet PPO achieves lower performance. This also demonstrates the effectiveness of GROW.

For reward normalization, we compared GROW with a trajectory-advantage baseline that computes relative advantages at the trajectory level. Given the limited rebuttal period, both methods were trained from the same checkpoint for 120 steps, with approximately matched total GPU-hours.

\hline & \text{Embodied Tasks} & \text{GUI Tasks} & \text{Combat Tasks} \\\\ \begin{array}{l} \text{Method} \\\\ \hline \text{Trajectory-advantage} \\\\ \text{GROW} \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 149 & 49.9 \pm 40.2 \\\\ 146 & 47.7 \pm 41.5 \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 324 & 20.4 \pm 26.0 \\\\ 305 & 36.5 \pm 31.5 \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 241 & 34.7 \pm 39.4 \\\\ 214 & 34.7 \pm 35.9 \end{array} \\\\ \hline \end{array}$$ GROW achieves higher success rates on GUI tasks and comparable performance on Combat and Embodied tasks, while using fewer interaction steps on successful episodes across all three categories. # 2. Response to Weakness 2 and Question: Limitations of the Surrogate Analysis and Potential Optimization Biases We thank the reviewer for the opportunity to clarify our surrogate analysis. ## 2.1 The Surrogate Analysis Shows That GROW Retains Trajectory-Level Optimization Signals, Rather Than Sharing the Optimization Direction of Trajectory-Level GRPO Under the assumption of approximately equal trajectory lengths, GROW retains a trajectory-level optimization signal through $\mathcal{J}_{\mathrm{traj}}$, which compares trajectories within the same rollout group, reinforcing those that outperform the baseline while suppressing those that fall below it. Meanwhile, $\mathcal{J}_{\mathrm{step}}$ captures the effects of temporal discounting, encouraging more efficient execution. This does not imply that GROW and trajectory-level GRPO share the same optimization direction. Since our surrogate analysis does not aim to approximate the objective of trajectory-level GRPO, differences in optimization direction arising from the use of action-wise probability ratios do not constitute a bias in the approximation we seek to establish. Nevertheless, we acknowledge that variations in trajectory length may affect the accuracy of the approximate decomposition of GROW’s objective into $\mathcal{J}_{\mathrm{traj}}$ and $\mathcal{J}_{\mathrm{step}}$. We analyze this effect in detail below.

2.2 Analysis of GROW under Unequal Trajectory Lengths

The approximately equal trajectory-length assumption in our surrogate analysis is introduced to simplify the derivation and highlight the trajectory-level optimization signals retained by GROW. To further examine the generality of this analysis, we extend the derivation without relying on this assumption. The resulting objective retains the original decomposition with an additional correction term accounting for trajectory-length variations.

2.2.1 Derivation without Relying on the Equal Trajectory-Length Assumption

Consider a group of GG trajectories, where trajectory ii has length HiH_i and terminal reward RiR_i. For step t∈1,…,Hit \in \\{1,\ldots,H_i\\}, define the discount factor and discounted reward as

\qquad r_{i,t}=d_{i,t}R_i.$$ The reward mean and normalized advantage across all state-action samples in the group are $$\mu \= \frac{\sum_{i=1}^{G}\sum_{t=1}^{H_i}d_{i,t}R_i} {\sum_{i=1}^{G}H_i}, \qquad A_{i,t} \= \frac{d_{i,t}R_i-\mu}{\sigma}.$$ Define the average discount factor for each trajectory and the group-level means as $$C_i=\frac{1}{H_i}\sum_{t=1}^{H_i}d_{i,t}, \qquad \bar C=\frac{1}{G}\sum_{i=1}^{G}C_i, \qquad s=\frac{1}{G}\sum_{i=1}^{G}R_i, \qquad \bar H=\frac{1}{G}\sum_{i=1}^{G}H_i.$$ Without imposing any equality constraint on $H_i$, we have the exact identity $$\begin{aligned} d_{i,t}R_i-\mu &= \bar C(R_i-s) + (d_{i,t}-\bar C)R_i - (\mu-\bar C s)\\\\ &= \bar C(R_i-s) + (d_{i,t}-\bar C)R_i - \Delta, \end{aligned}$$ where $\Delta=\mu-\bar C s$. Therefore, $$A_{i,t} \= \frac{1}{\sigma} \left[ \bar C(R_i-s) + (d_{i,t}-\bar C)R_i - \Delta \right].$$ Then, following the main paper, we omit clipping and substitute this decomposition into the objective to obtain: $$\mathcal{J} \= \mathcal{J}_{\mathrm{traj}} +\mathcal{J}_{\mathrm{step}} -\mathcal{J}_{\Delta},$$ where the three components are defined as $$\begin{aligned} \mathcal{J}_{\mathrm{traj}} &= \underset{\substack{ c\sim\mathcal{C}\\\\ a_{i,t}\sim\pi_{\mathrm{old}}(\cdot\mid s_{i,t},c) }}{\mathbb{E}} \left[ \frac{\bar C}{\sigma} \frac{1}{G}\sum_{i=1}^{G}\frac{1}{H_i} \sum_{t=1}^{H_i} \rho_{i,t}(R_i-s) \right],\\\\[6pt] \mathcal{J}_{\mathrm{step}} &= \underset{\substack{ c\sim\mathcal{C}\\\\ a_{i,t}\sim\pi_{\mathrm{old}}(\cdot\mid s_{i,t},c) }}{\mathbb{E}} \left[ \frac{1}{\sigma} \frac{1}{G}\sum_{i=1}^{G}\frac{1}{H_i} \sum_{t=1}^{H_i} \rho_{i,t}(d_{i,t}-\bar C)R_i \right],\\\\[6pt] \mathcal{J}_{\Delta} &= \underset{\substack{ c\sim\mathcal{C}\\\\ a_{i,t}\sim\pi_{\mathrm{old}}(\cdot\mid s_{i,t},c) }}{\mathbb{E}} \left[ \frac{1}{\sigma} \frac{1}{G}\sum_{i=1}^{G}\frac{1}{H_i} \sum_{t=1}^{H_i} \rho_{i,t}\Delta \right]. \end{aligned}$$ Even with unequal trajectory lengths within a rollout group, GROW’s objective retains $\mathcal{J}_{\mathrm{traj}}$ and $\mathcal{J}_{\mathrm{step}}$, alongside a correction term $\mathcal{J}_{\Delta}$ that may affect the optimization direction. We quantify this effect empirically below. ### 2.2.2 Quantitative Analysis Shows That $\mathcal{J}_\Delta$ Has a Limited Impact on the Optimization Direction Following the reviewer's suggestion, we quantify the effect of the correction term $\mathcal{J}_\Delta$ induced by unequal trajectory lengths on the optimization direction. We use the same training tasks as in the main paper and sample 45 rollout groups, computing the cosine similarity for 27 pairs of gradients by comparing the gradients of the full surrogate objective and a reference objective with the correction term removed. | Metric | Mean ± standard deviation | |---|---:| | Gradient cosine similarity | 0.9880 ± 0.0096 | The high cosine similarity indicates that the two gradients remain closely aligned, supporting a limited effect of $\mathcal{J}_\Delta$ on the optimization direction in this comparison. ### 2.2.3 GROW Assigns Equal Total Loss Weight to Trajectories Regardless of Their Lengths Equation 4 already includes the factor $1/H_i$ to average the loss over the state-action samples within each trajectory. Therefore, longer trajectories do not receive greater total loss weight simply because they contain more samples.

I appreciate the authors' theoretical derivations and the additional experimental results provided. I have also carefully reviewed the questions raised by other reviewers and the authors' responses, which have, to a certain extent, addressed my concerns regarding "trajectory length." Could the authors provide further clarification on the "gradient cosine similarity" experiment—specifically, which tasks were used for this evaluation, and how do the results vary across different tasks?

Reviewer_C37ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GROW splits interaction trajectories into state–action samples to reduce the context cost of VLM reinforcement learning. It assigns discounted terminal rewards to individual steps and computes relative advantages across steps within each rollout group. Experiments cover Minecraft, simulated Language Table, and WebShop.

Critical strengths

The method is simple and does not require a value network. Starting from the same policy, it improves Minecraft GUI task success from 43.9% with PPO to 68.4%, while reducing completion steps. RL training on just eight tasks also improves performance on tasks not used for RL, suggesting some transfer of learned skills.

Critical weaknesses
  1. The cost of losing history is not sufficiently studied. The policy mainly uses the current image and task instruction, which saves resources but leaves out past observations. A single image may not reveal where the agent has already explored or how a target was moving. Long interactions do not necessarily require long-term memory. The paper needs memory-dependent tasks and comparisons with different history lengths to establish its ability to handle long-term decisions.

  2. Temporal discounting does not necessarily assign credit correctly. Early but essential actions in successful trajectories may receive negative advantages, while useful actions in failed trajectories may be penalized. When all rollouts fail, the method still provides no useful learning signal. The analysis assumes roughly equal trajectory lengths and omits clipping, so it does not resolve these concerns. Experiments with delayed rewards, important early actions, and low success rates would help.

  3. The comparisons do not fully explain the performance and efficiency gains. The trajectory-level GRPO baseline often reaches its 32K context limit before completing tasks and therefore misses terminal rewards. This makes it hard to separate better optimization from simply allowing longer interactions. Beyond the existing discount-factor ablation, comparisons with sliding-window or compressed-context GRPO under the same interaction budget are needed. The paper also lacks direct comparisons of peak memory, training time, throughput, and GPU-hours. Fewer steps to finish a task do not establish lower training costs.

Questions for authors

see Weakness.

4 comments

1. Response to Weakness 1: Limitation of Lack of Long-Term Memory

We thank the reviewer for raising this concern. As described in Section 3.1, GROW focuses on open-world tasks formulated as Markov decision processes, rather than tasks requiring long-term memory for decision-making. Making decisions without relying on long-term memory is also a common practice among open-world agents, including JARVIS-VLA, OpenHA, and MAIN-VLA, as listed in Table 1 of the main paper. We agree that long interactions do not necessarily demonstrate long-term memory and that some tasks require information unavailable in the current observation. Our current method cannot handle such memory-dependent tasks, and addressing this limitation is a direction for future research.

2. Response to Weakness 2: Temporal Discounting Can Provide Effective Learning Signals for Optimization

We thank the reviewer for raising these concerns. We agree that temporal discounting does not guarantee correct credit assignment for every action. Our analysis cannot sufficiently resolve these concerns. We therefore provide empirical evidence for important early actions and a detailed explanation below.

2.1 Important Early Actions Retain Positive Relative Advantages with an Appropriately Large Discount Factor

Important early actions are associated with successful trajectories and can retain positive relative advantages when the discount factor is appropriately large. To test this, we examined opening the GUI, a prerequisite for crafting and smelting whose reward is often delayed. We measured its mean relative advantage over more than 30 rollout groups under different discount factors. The table below presents these measurements alongside the GUI task success rates from Table 4 in the main paper.

γMean relative advantage for opening the GUIGUI task average success rate
0.995+0.162868.4 ± 29.0%
0.95+0.120862.0 ± 20.6%
0.9+0.029634.9 ± 34.4%

The positive mean advantages show that this important early action remains encouraged on average. Stronger discounting reduces this mean relative advantage and coincides with lower task success, whereas our chosen γ = 0.995 preserves a larger positive advantage and achieves higher success. These results demonstrate that even though temporal discounting does not necessarily assign credit correctly, GROW can effectively learn important early actions.

2.2 The Lack of Learning Signals from All-Failure Rollouts and the Potential Penalization of Useful Actions Are Shared Limitations of Sparse-Reward GRPO Methods

The absence of effective learning signals when all rollouts fail and the potential penalization of useful actions in failed trajectories are shared limitations of sparse-reward GRPO methods. GROW does not claim to solve these limitations. Instead, its contribution is to decompose trajectories into state-action samples, apply temporal discounting for credit assignment, and compute relative advantages across these samples within the same rollout group. This allows GROW to optimize state-action samples from complete rollouts, enabling training on open-world tasks where trajectory samples would exceed the context limit and place substantial demands on training resources.

3. Response to Weakness 3: Comparisons to Explain the Performance and Efficiency Gains

We thank the reviewer for raising this concern. Following the suggestion, we added a sliding-window GRPO comparison under the same environment interaction budget and direct measurements of training resources.

3.1 GROW Outperforms Sliding-Window GRPO under the Same Interaction Budget

Given the limited rebuttal period, we trained GROW and sliding-window GRPO from the same checkpoint for 120 training steps under the same environment interaction budget. The table reports their evaluation results.

\hline & \text{Embodied Tasks} & \text{GUI Tasks} & \text{Combat Tasks} \\\\ \begin{array}{l} \text{Method} \\\\ \hline \text{Sliding-window GRPO} \\\\ \text{GROW} \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 192 & 30.1 \pm 37.1 \\\\ 146 & 47.7 \pm 41.5 \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 349 & 4.3 \pm 12.3 \\\\ 305 & 36.5 \pm 31.5 \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 255 & 26.7 \pm 35.3 \\\\ 214 & 34.7 \pm 35.9 \end{array} \\\\ \hline \end{array}$$ GROW achieves higher success rates and uses fewer interaction steps across all three task categories under the same interaction budget. ## 3.2 Direct Comparisons Show That GROW Achieves Lower Peak GPU Memory In our paper, reducing the demands on training resources primarily refers to the lower peak GPU memory usage achieved by GROW when training on state-action samples decomposed from complete trajectories, thereby avoiding GPU out-of-memory errors. We will clarify this point with more precise wording in the revised manuscript. Under the same environment interaction budget, GROW optimizes all state–action samples from complete rollout groups, whereas sliding-window GRPO optimizes incomplete 32K-token trajectory samples constructed using a sliding window. Although GROW takes longer per training step and requires more GPU hours, it achieves lower peak GPU memory usage and 2.87× higher training token throughput than sliding-window GRPO. This enables GROW to optimize all state-action samples from complete rollout groups. In contrast, sliding-window GRPO, which applies a sliding window to trajectory-level GRPO, optimizes only samples from partial trajectories with context lengths limited to 32K, yet requires higher peak GPU memory and achieves lower performance than GROW. Furthermore, directly optimizing complete trajectory samples with trajectory-level GRPO, whose context lengths can exceed 64K tokens, as shown in Figure 1 of the main paper, would require even more peak GPU memory and lead to GPU out-of-memory errors. | Metric | 32K sliding-window GRPO | GROW | |---|---:|---:| | Peak per-GPU memory, GiB | 95.23 | 90.59 | | Average training time per step, minutes | 18.31 | 25.59 | | Training throughput, tokens/sec/GPU | 229.13 | 657.88 | | GPU Hours | 586 | 819 |

Thank you for the response, which has addressed most of my concerns. I will maintain my original score.

AuthorsOpenReview ↗

Thanks to Reviewer C37R

Thank you very much for your careful review and constructive feedback. We truly appreciate your time, effort, and thoughtful consideration of our work!

Reviewer_pKVHOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a reinforcement learning framework, GROW, to extend the original GRPO method. The motivation behind GROW is that the lengths of trajectory samples are too long, which means consuming too many training resources to train VLM agents in the open-world practically. GROW addresses this issue by decomposing the trajectory samples into state-action samples. Then GROW performs credit assignment and computes advantages between state-action samples of the same rollout groups to optimize the policy effectively. This paper also provides theoretical analysis to show that GROW still optimizes the policy with trajectory-level training signals while avoiding directly computing with trajectory samples with excessively long contexts. In empirical experiments, GROW achieves SOTA performance on more than 800 Minecraft tasks.

Critical strengths
  1. The superior performance of GROW is substantiated by theoretical analysis. The surrogate analysis shows that even though trajectory samples are not used to compute the objective directly, the objective can still contain trajectory-level optimization signals, which solidifies the claim.
  2. The evaluation results in Minecraft show GROW improves the average success rate and steps needed to finish tasks. In addition, the superior performance on OOD tasks demonstrates the generalization and skill transferability of the policy trained by GROW. Additionally, this paper extends GROW to other environments, showing that the effectiveness of GROW generalizes beyond Minecraft.
Critical weaknesses

Unlike standard GRPO, which computes advantages among rollouts conditioned on the same prompts to estimate the quality of different rollouts, GROW normalizes rewards across different contexts and time steps. However, the theoretical meaning of these advantages does not seem to be well explained.

Questions for authors
  1. Could the authors analyze the advantage function to clarify its theoretical meaning?
  2. In Section 3.2, this paper seems to assume the standard deviation is not zero, however, when all trajectories succeed or fail, the standard deviation will be zero. How does GROW handle zero-variance groups when all trajectories within a rollout group either succeed or fail?
  3. The discounted reward substantially upweights actions near the end of successful trajectories. How does GROW avoid weakening the learning signal for crucial early actions that are far from task completion?
3 comments

1. Response to Weakness and Question 1: Theoretical meaning of the advantage function

We thank the reviewer for the question. Although the advantage function normalizes the discounted reward for different states, Equation 6 shows that the advantage function is approximately equal to γHi−tR(τi)−CγS\gamma^{H_i - t}\mathcal{R}(\tau_i) - C_{\gamma}{S}, which can be rewritten as Cγ(R(τi)−S)+(γHi−t−Cγ)R(τi)C_{\gamma}(\mathcal{R}(\tau_i) - S) + (\gamma^{H_i - t} - C_\gamma)\mathcal{R}(\tau_i). Therefore, the advantages can be interpreted from two perspectives. First, it can estimate the relative advantages of different rollout trajectories. Second, the second term assigns larger advantages to state-action samples that are closer to success, thereby encouraging the agent to complete the task more efficiently. In summary, the theoretical meaning of the advantage function is that it estimates the quality of the rollout trajectory to which the state-action samples belong and encourages the agent to complete tasks efficiently.

2. Response to Question 2: Handling Zero-Variance Groups

We thank the reviewer for pointing out this issue. An all-success group does not generally have zero variance in GROW: due to temporal discounting, its state-action rewards are ri,t=γHi−tr_{i,t}=\gamma^{H_i-t}, which vary across timesteps when γ<1\gamma<1. Thus, successful trajectories can still provide nonzero relative learning signals that emphasize actions closer to completion. In contrast, when all rollouts fail, every state-action reward is zero, yielding zero variance and zero advantages. We skip these groups because they provide no policy-gradient learning signal.

3. Response to Question 3: Learning signal for early important actions

We thank the reviewer for raising this concern. We provide further analysis and experimental evidence to address this concern.

3.1 Temporal Discounting Preserves Effective Learning Signals for Important Early Actions

Although temporal discounting reduces the reward assigned to early actions, an effective early action in a successful trajectory still receives a non-negligible discounted reward. Across a sufficient number of rollouts, such actions consistently occur in successful trajectories and therefore obtain stable relative advantages, which reinforces their behavior. We select γ\gamma to balance execution efficiency and long-horizon credit assignment: a sufficiently large γ\gamma ensures that important early actions retain reasonable rewards and reliable learning signals.

3.2 Experimental Results Confirm That Important Early Actions Receive Positive Learning Signals

To examine whether GROW provides effective learning signals for important early actions, we analyze the action of opening the GUI, a prerequisite for crafting and smelting tasks with delayed terminal rewards. We measure the mean relative advantage of this action across more than 30 rollout groups under different discount factors. The results are presented below, together with the GUI task success rates from Table 4.

γ\gammaMean Relative Advantage of Opening the GUIGUI Task Success Rate
0.995+0.162868.4 ± 29.0%
0.95+0.120862.0 ± 20.6%
0.9+0.029634.9 ± 34.4%

As shown in the table above, with γ=0.995\gamma=0.995, opening the GUI receives the largest mean advantage, and the success rate on GUI tasks is the highest. Lowering γ\gamma strengthens temporal discounting and is accompanied by smaller mean advantages and lower success rates. The mean advantages remain positive in every setting, indicating that this action is still encouraged on average. Together, these observations suggest that limitations in credit assignment do not prevent GROW from effectively learning important early actions.

Thank you for your response and clarifications. After considering the rebuttal, I have decided to retain my original score.

AuthorsOpenReview ↗

Thanks to Reviewer pKVH

Thank you again for your valuable feedback and thoughtful review. We sincerely appreciate your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

We sincerely thank you for your time and careful consideration of this statement. We also thank the reviewers for their careful reviews and valuable feedback.

1. Strengths

The reviews recognize the following strengths of our submission:

  • Simple and Efficient Design. GROW requires no value network and reduces training context length, offering a practical solution for long-horizon VLM agent training [C37R, sqVV].

  • Theoretical Support. The surrogate analysis shows that GROW's objective can retain trajectory-level optimization signals without directly using trajectory samples to compute the objective [pKVH].

  • Empirical Performance and Generalization. Evaluation on over 800 Minecraft tasks shows higher success rates and fewer completion steps. Gains on OOD tasks suggest skill transfer from RL training on just eight tasks, while Language Table and WebShop experiments provide preliminary evidence of applicability beyond Minecraft [pKVH, C37R, sqVV].


2. Clarifications

We addressed the reviewers’ main concerns through the rebuttal and discussion, as summarized below.

Reviewer pKVH

  • Feedback: The reviewer questioned the theoretical meaning of the advantage, the handling of zero-variance groups, and whether temporal discounting weakens learning signals for important early actions.

  • Our Response: We explained that the advantage reflects relative trajectory quality and encourages efficient completion. We clarified that discounting can preserve learning signals in all-success groups, while all-failure groups are skipped. Analysis of opening the GUI, an early prerequisite action, found positive mean advantages across discount factors, with γ=0.995\gamma=0.995 yielding the strongest signal and highest GUI success rate.

Reviewer C37R

  • Feedback: The reviewer questioned the impact of omitting history and the reliability of temporal credit assignment, particularly for early actions and failed rollouts. They also requested GRPO comparisons under equal interaction budgets and direct measurements of training memory, time, throughput, and GPU hours.

  • Our Response: GROW targets open-world MDPs. Making decisions without relying on long-term memory is common practice among open-world agents. We show positive mean advantages for opening the GUI and outperform sliding-window GRPO under equal interaction budgets, with lower memory usage and higher throughput.

Reviewer sqVV

  • Feedback: The reviewer questioned GROW's novelty, requested ablations, and asked whether unequal trajectory lengths and action-wise ratios distort weighting or undermine the theoretical claims. The reviewer also requested the evaluated tasks and gradient cosine similarities for each task.

  • Our Response: We clarified GROW's novelty and its differences from trajectory-level GRPO, added a normalization ablation, and extended the analysis to unequal lengths with equal trajectory weighting. We also listed the evaluated tasks and reported consistently high gradient cosine similarities for each.


3. Rebuttal and Discussion Summary

  • Reviewer pKVH acknowledged our response and clarifications and retained the original score.

  • Reviewer C37R confirmed that our response addressed most concerns and maintained the original score.

  • Reviewer sqVV responded positively to our rebuttal and maintained the original score.

In summary, we have addressed all of the major concerns in the official reviews and strengthened the paper’s empirical evidence and clarity.


We respectfully hope that the final evaluation will take into account both the paper's contributions and the fact that the major concerns raised in the reviews have been addressed. Thank you again for your time and consideration.

Best wishes,
The Authors of Submission 11331

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.