- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GROW, which aims to address the challenges—namely, excessive context length and high computational resource consumption when training open-world Vision-Language Model (VLM) agents using Group Relative Policy Optimization (GRPO) on long interaction trajectories. GROW decomposes full trajectories into individual state-action samples. It propagates sparse, trajectory-level rewards to these individual samples via temporal discounting and then calculates relative advantages within each rollout group to optimize the policy. The authors provide a surrogate analysis, arguing that under certain assumptions, this approach preserves the optimization signals present at the trajectory level while avoiding the need to train directly on full trajectories. Experiments were primarily conducted on a Minecraft benchmark comprising over 800 tasks, spanning embodied manipulation, GUI interaction, and combat. The results show that GROW achieves success rates of 59.6%, 68.4%, and 49.0% across these three categories, respectively, demonstrating superior overall performance compared to trajectory-level GRPO and PPO. The authors also conducted extended experiments in simulated environments, including Language Table and WebShop. Overall, the paper addresses a problem of practical significance, proposes a relatively straightforward method, and reports positive experimental results. However, the degree of independent innovation in the core algorithm—relative to existing policy optimization methods—as well as the strength of the theoretical analysis in explaining the actual optimization objectives, still need further validation.
Critical strengths
- It offers a simple yet promising solution to a practical bottleneck in training long-horizon VLM agents. Open-world tasks typically require multiple rounds of visual observation and action interaction; the cumulative context of a full trajectory imposes significant demands on memory and computational resources. By decomposing training data into individual state-action samples, GROW drastically reduces the context length per training sample. The paper reports that the context length for GROW’s state-action samples is under 1K tokens, whereas trajectory-level GRPO can reach up to 32K tokens. This design offers clear engineering value and provides a practical pathway for reinforcement learning in long-horizon agent training.
- The experimental scope is extensive and demonstrates compelling performance improvements. The authors evaluated GROW across more than 800 Minecraft tasks, compared it against trajectory-level GRPO, PPO, and initialization policies, and conducted evaluations on both in-distribution (ID) and out-of-distribution (OOD) generalization. GROW achieved high success rates across embodied, GUI, and combat tasks; notably, in GUI tasks, the success rate increased from 43.9% (with PPO) to 68.4%. Furthermore, the authors conducted additional experiments on Language Table and WebShop, providing preliminary evidence that the method may be applicable to various interactive environments.
Critical weaknesses
- The core algorithmic innovation lacks sufficient novelty, and the fundamental advantages of GROW over existing policy optimization methods have not been clearly established. The primary design of GROW involves distributing trajectory-level rewards to individual state-action samples via temporal discounting, followed by group-wise reward normalization and advantage calculation. Structurally, this approach is closely related to existing methods based on reward shaping, temporal discounting, and local policy gradient optimization. Although the paper introduces a "surrogate analysis," it remains unclear whether this analysis reveals a mechanism that is substantively distinct from conventional reward propagation and policy optimization techniques. Specifically, GROW calculates advantages at the level of individual state-action samples, whereas standard GRPO typically operates at the level of complete responses or trajectories. This distinction may simultaneously alter reward normalization, sample weighting, and gradient estimation. Consequently, the current results do not sufficiently clarify whether the performance gains stem from the contextual advantages provided by trajectory decomposition, or from the specific modifications to reward propagation and sample weighting. It is recommended to include more granular ablation studies—for instance, by isolating the effects of temporal discounting, varying the reward normalization method, and comparing different optimization strategies while keeping the computational budget constant.
- The theoretical analysis relies on strong, simplifying assumptions, which are insufficient to fully substantiate the authors' claims regarding the preservation of optimization signals at the trajectory level. The authors' surrogate analysis assumes that trajectory lengths within the same rollout group are approximately equal; based on this, they decompose the GROW optimization objective into trajectory-level signals and terms related to temporal discounting. However, in open-world tasks, trajectory lengths can vary significantly—with successful and failed trajectories often exhibiting different length distributions. In such scenarios, the mean and variance calculated across individual state-action samples may be skewed by trajectory lengths, thereby altering the relative weighting of different trajectories and their constituent actions. Furthermore, while GROW computes policy probability ratios for each state-action sample independently, trajectory-level policy updates involve the joint probability of the entire trajectory. The authors need to clarify the conditions under which their current objective effectively approximates the optimization direction of trajectory-level GRPO, and quantify the potential bias introduced by variations in trajectory length and the use of action-wise probability ratios. The current analysis serves more as an interpretive decomposition of the objective function rather than a rigorous proof that GROW preserves the optimization properties of trajectory-level GRPO in the general case.
Questions for authors
The core claim of the paper is that GROW retains the essential optimization signals of trajectory-level GRPO while avoiding the need to train directly on full trajectories. Could the authors clarify: when trajectory lengths within a rollout group vary significantly, does the uniform calculation of reward means, standard deviations, and relative advantages across all state-action samples lead to disproportionately high optimization weights for longer trajectories—thereby shifting the optimization direction away from that of the original trajectory-level GRPO?
6 comments
2.2 Analysis of GROW under Unequal Trajectory Lengths
The approximately equal trajectory-length assumption in our surrogate analysis is introduced to simplify the derivation and highlight the trajectory-level optimization signals retained by GROW. To further examine the generality of this analysis, we extend the derivation without relying on this assumption. The resulting objective retains the original decomposition with an additional correction term accounting for trajectory-length variations.
2.2.1 Derivation without Relying on the Equal Trajectory-Length Assumption
Consider a group of trajectories, where trajectory has length and terminal reward . For step , define the discount factor and discounted reward as
\qquad r_{i,t}=d_{i,t}R_i.$$ The reward mean and normalized advantage across all state-action samples in the group are $$\mu \= \frac{\sum_{i=1}^{G}\sum_{t=1}^{H_i}d_{i,t}R_i} {\sum_{i=1}^{G}H_i}, \qquad A_{i,t} \= \frac{d_{i,t}R_i-\mu}{\sigma}.$$ Define the average discount factor for each trajectory and the group-level means as $$C_i=\frac{1}{H_i}\sum_{t=1}^{H_i}d_{i,t}, \qquad \bar C=\frac{1}{G}\sum_{i=1}^{G}C_i, \qquad s=\frac{1}{G}\sum_{i=1}^{G}R_i, \qquad \bar H=\frac{1}{G}\sum_{i=1}^{G}H_i.$$ Without imposing any equality constraint on $H_i$, we have the exact identity $$\begin{aligned} d_{i,t}R_i-\mu &= \bar C(R_i-s) + (d_{i,t}-\bar C)R_i - (\mu-\bar C s)\\\\ &= \bar C(R_i-s) + (d_{i,t}-\bar C)R_i - \Delta, \end{aligned}$$ where $\Delta=\mu-\bar C s$. Therefore, $$A_{i,t} \= \frac{1}{\sigma} \left[ \bar C(R_i-s) + (d_{i,t}-\bar C)R_i - \Delta \right].$$ Then, following the main paper, we omit clipping and substitute this decomposition into the objective to obtain: $$\mathcal{J} \= \mathcal{J}_{\mathrm{traj}} +\mathcal{J}_{\mathrm{step}} -\mathcal{J}_{\Delta},$$ where the three components are defined as $$\begin{aligned} \mathcal{J}_{\mathrm{traj}} &= \underset{\substack{ c\sim\mathcal{C}\\\\ a_{i,t}\sim\pi_{\mathrm{old}}(\cdot\mid s_{i,t},c) }}{\mathbb{E}} \left[ \frac{\bar C}{\sigma} \frac{1}{G}\sum_{i=1}^{G}\frac{1}{H_i} \sum_{t=1}^{H_i} \rho_{i,t}(R_i-s) \right],\\\\[6pt] \mathcal{J}_{\mathrm{step}} &= \underset{\substack{ c\sim\mathcal{C}\\\\ a_{i,t}\sim\pi_{\mathrm{old}}(\cdot\mid s_{i,t},c) }}{\mathbb{E}} \left[ \frac{1}{\sigma} \frac{1}{G}\sum_{i=1}^{G}\frac{1}{H_i} \sum_{t=1}^{H_i} \rho_{i,t}(d_{i,t}-\bar C)R_i \right],\\\\[6pt] \mathcal{J}_{\Delta} &= \underset{\substack{ c\sim\mathcal{C}\\\\ a_{i,t}\sim\pi_{\mathrm{old}}(\cdot\mid s_{i,t},c) }}{\mathbb{E}} \left[ \frac{1}{\sigma} \frac{1}{G}\sum_{i=1}^{G}\frac{1}{H_i} \sum_{t=1}^{H_i} \rho_{i,t}\Delta \right]. \end{aligned}$$ Even with unequal trajectory lengths within a rollout group, GROW’s objective retains $\mathcal{J}_{\mathrm{traj}}$ and $\mathcal{J}_{\mathrm{step}}$, alongside a correction term $\mathcal{J}_{\Delta}$ that may affect the optimization direction. We quantify this effect empirically below. ### 2.2.2 Quantitative Analysis Shows That $\mathcal{J}_\Delta$ Has a Limited Impact on the Optimization Direction Following the reviewer's suggestion, we quantify the effect of the correction term $\mathcal{J}_\Delta$ induced by unequal trajectory lengths on the optimization direction. We use the same training tasks as in the main paper and sample 45 rollout groups, computing the cosine similarity for 27 pairs of gradients by comparing the gradients of the full surrogate objective and a reference objective with the correction term removed. | Metric | Mean ± standard deviation | |---|---:| | Gradient cosine similarity | 0.9880 ± 0.0096 | The high cosine similarity indicates that the two gradients remain closely aligned, supporting a limited effect of $\mathcal{J}_\Delta$ on the optimization direction in this comparison. ### 2.2.3 GROW Assigns Equal Total Loss Weight to Trajectories Regardless of Their Lengths Equation 4 already includes the factor $1/H_i$ to average the loss over the state-action samples within each trajectory. Therefore, longer trajectories do not receive greater total loss weight simply because they contain more samples.I appreciate the authors' theoretical derivations and the additional experimental results provided. I have also carefully reviewed the questions raised by other reviewers and the authors' responses, which have, to a certain extent, addressed my concerns regarding "trajectory length." Could the authors provide further clarification on the "gradient cosine similarity" experiment—specifically, which tasks were used for this evaluation, and how do the results vary across different tasks?
1. Response to Weakness 1: Insufficient Algorithmic Novelty and Lack of Comprehensive Ablation Studies
We thank the reviewer for these comments. We clarify GROW's novelty and address the suggested ablations below.
1.1 GROW's Novelty Lies in Effectively Optimizing Trajectory Samples via State-Action Decomposition for Open-World Tasks
Although methods like temporal discounting have been used in prior work, GROW's novelty lies in decomposing trajectories into state-action samples and allocating trajectory-level rewards to these samples through temporal discounting. This enables GROW to optimize state-action samples from complete rollouts and train on open-world tasks whose full trajectory samples would otherwise exceed the context limit and thus impose substantial demands on training resources.
1.2 Ablations of Temporal Discounting, Reward Normalization, and Optimization Strategies Validate GROW's Effectiveness
The reviewer suggests three types of ablations. For temporal discounting, Table 4 in the main paper already reports results with γ = 1.0, which removes discounting. Since both settings use the same number of training steps and have similar GPU-hours per training step, their total GPU-hours are approximately equal.
For optimization strategies, Table 3 evaluates PPO applied to state-action samples. Each PPO training step takes longer than a GROW training step, resulting in higher GPU-hours under the same number of training steps, yet PPO achieves lower performance. This also demonstrates the effectiveness of GROW.
For reward normalization, we compared GROW with a trajectory-advantage baseline that computes relative advantages at the trajectory level. Given the limited rebuttal period, both methods were trained from the same checkpoint for 120 steps, with approximately matched total GPU-hours.
\hline & \text{Embodied Tasks} & \text{GUI Tasks} & \text{Combat Tasks} \\\\ \begin{array}{l} \text{Method} \\\\ \hline \text{Trajectory-advantage} \\\\ \text{GROW} \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 149 & 49.9 \pm 40.2 \\\\ 146 & 47.7 \pm 41.5 \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 324 & 20.4 \pm 26.0 \\\\ 305 & 36.5 \pm 31.5 \end{array} & \begin{array}{rr} \text{Steps} & \text{ASR} \\\\ \hline 241 & 34.7 \pm 39.4 \\\\ 214 & 34.7 \pm 35.9 \end{array} \\\\ \hline \end{array}$$ GROW achieves higher success rates on GUI tasks and comparable performance on Combat and Embodied tasks, while using fewer interaction steps on successful episodes across all three categories. # 2. Response to Weakness 2 and Question: Limitations of the Surrogate Analysis and Potential Optimization Biases We thank the reviewer for the opportunity to clarify our surrogate analysis. ## 2.1 The Surrogate Analysis Shows That GROW Retains Trajectory-Level Optimization Signals, Rather Than Sharing the Optimization Direction of Trajectory-Level GRPO Under the assumption of approximately equal trajectory lengths, GROW retains a trajectory-level optimization signal through $\mathcal{J}_{\mathrm{traj}}$, which compares trajectories within the same rollout group, reinforcing those that outperform the baseline while suppressing those that fall below it. Meanwhile, $\mathcal{J}_{\mathrm{step}}$ captures the effects of temporal discounting, encouraging more efficient execution. This does not imply that GROW and trajectory-level GRPO share the same optimization direction. Since our surrogate analysis does not aim to approximate the objective of trajectory-level GRPO, differences in optimization direction arising from the use of action-wise probability ratios do not constitute a bias in the approximation we seek to establish. Nevertheless, we acknowledge that variations in trajectory length may affect the accuracy of the approximate decomposition of GROW’s objective into $\mathcal{J}_{\mathrm{traj}}$ and $\mathcal{J}_{\mathrm{step}}$. We analyze this effect in detail below.