acceptodds
Under review as a conference paper at ICLR 2027

GTPO: Group Teaching Policy Optimization via Temporal Optimal Transport in VLA Reinforcement Learning

Abstract

Reinforcement learning can improve vision-language-action policies beyond their demonstrations, but costly robot interactions and sparse terminal feedback make this process inefficient. Reliable dense supervision could extract more useful information from each interaction, yet learned reward models often require substantial additional data construction and training. Can the policy's own successful and failed rollouts provide this supervision? We propose Group Teaching Policy Optimization (GTPO), which turns successful rollouts into teachers for failed attempts at the same task from similar initial states. Building on Group Relative Policy Optimization (GRPO) and drawing inspiration from inverse reinforcement learning, GTPO uses trajectory comparison to refine otherwise uniform trajectory-level advantages. Each failed rollout is paired with its nearest successful in-group teacher using the policy's visual features. Temporally masked optimal transport aligns their trajectories and produces normalized action-chunk weights that redistribute the original GRPO advantage while retaining terminal rewards. The teachers evolve with the policy, requiring no additional expert reference trajectories or separately trained reward model for teaching. We evaluate GTPO on LIBERO, RoboTwin, and five challenging real-world tasks requiring fine manipulation. Across four LIBERO suites, GTPO achieves 98.65% average success with one demonstration per task and 99.70% with the full demonstration set, requiring at least 22.2% fewer aggregate training steps to reach 95% success than matched GRPO. On the five real-world tasks, GTPO achieves 80–100% success after three on-policy rounds, exceeding GRPO by 10–35 percentage points. Project website: https://gtpo-project.github.io/gtpo_page/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Reinforcement Learning, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x2jgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GTPO, RL approach for vision-language action policies that improves trajectory-level GRPO advantages using successful rollouts as teachers. For each failed trajectory method selects the closest successful rollout, computes a temporally constrained optimal transport alignment over visual features, and uses the resulting costs to redistribute negative advantages across actions.

Critical strengths
  • The paper addresses an important problem: extracting informative temporal supervision from sparse terminal rewards without training an additional model.
  • The method achieves improvements over GRPO on LIBERO and RoboTwin, with the largest gains from the one-demonstration warm start.
  • The method improves training efficiency (AUC-SR, Early SR).
  • The evaluation includes real-world robotic manipulation tasks.
  • The appendix is detailed, and the limitations of visual comparison are stated.
Critical weaknesses

W1. No comparison with randomly shuffled GTPO weights. This comparison is necessary to distinguish meaningful temporal credit assignment from generic non-uniform advantage reweighting. A trajectory permutation keeps the mean weight at 1, so it matches the GRPO gradient in expectation while adding the same variance.

W2. Dependence of weight concentration on trajectory length. Since the transport plan has uniform row marginals 1/Li1/L_i, ∣si,t∣≤2/Li|s_{i,t}| \le 2/L_i, and the weight ratio is bounded by exp⁡(2/(Liτ))\exp(2/(L_i\tau)). Longer trajectories therefore have a higher effective softmax temperature τLi\tau L_i and more uniform credit weights, which reduces the ability to distinguish important actions. This matters most in the real-world setting, where failed prefixes are truncated at takeover and τ=0.01\tau = 0.01. In simulation (Li=39L_i = 39), the weights differ by at most about 5% once τ\tau reaches 1. App. B.3 acknowledges the scale issue but does not resolve it.

W3. Limited validation of action failure localization. Visual dissimilarity does not necessarily identify the failure-inducing action. An incorrect action may cause visual divergence at later stages, leading GTPO to penalize the following actions rather than the failure-inducing one; in Fig. 6 the weights peak several chunks after the annotated failed grasp. The authors acknowledge this limitation, but localization is not quantified, although simulation provides ground-truth failure events. The temporal mask is also nearly vacuous at the defaults (k=30k = 30 against L≤39L \le 39), and balanced OT may distort the alignment when trajectory lengths differ (an early-terminating teacher versus a failure that runs to the horizon, or a truncated real-world prefix). Unbalanced or partial OT, which does not force every chunk of the failure to be matched, may be more appropriate.

W4. Statistical support. The results appear to come from single training runs.

W5. Baselines GRPO is the only baseline trained under matched conditions. The closest related approach is graph-based credit assignment (GraphGPO, Cheng et al., ICML 2026), which merges shared prefixes of group rollouts into a state-transition graph and assigns credit at the branching points where trajectories diverge, i.e., where the failure-inducing action should be. Exact state merging does not apply to continuous visual observations, but an approximate variant (merging states within a feature-distance threshold, or concentrating negative credit at the first divergence from the teacher) would be a natural baseline and would directly test W3. TemporalOT used as a per-step reward or PPO/GAE would also clarify the contribution.

Questions for authors
  1. How does GTPO compare to randomly permuted weights, with three seeds for GTPO, permuted GTPO, and GRPO?
  2. Have the authors considered a row-normalized alignment cost, sˉi,t=Li si,t\bar s_{i,t} = L_i\, s_{i,t}, or a per-trajectory range normalization? How would this affect performance on trajectories of different lengths, e.g., the truncated real-world prefixes? What is the empirical distribution of wi,tw_{i,t} over training?
  3. How strongly do the OT-derived credit weights correlate with the actions actually responsible for failures, e.g., relative to simulator failure events?
  4. Why constant τ=1\tau = 1 (near-uniform weights) underperforms GRPO on TTS@95?
  5. In Table 4, the pointwise score −ctt(i,j)-c^{(i,j)}_{tt} lacks the transport mass 1/Li1/L_i contained in the OT score, so at the same τ\tau it is about LiL_i times larger and gives much sharper weights (as noted in App. B.3). Was it rescaled by 1/Li1/L_i? Otherwise the comparison conflates the alignment method with weight concentration.
5 comments
AuthorsOpenReview ↗

Response to Reviewer x2jg

Thank you for the detailed analysis of the weighting mechanism and the suggested controls. We have added the shuffled-weight comparison, a same-seed temperature comparison, and empirical weight statistics, and clarified the pointwise-score scaling.

[W1, W4 & Q1: Shuffled Weights and Independent Training Runs]

R1: We compared GRPO, shuffled-GTPO, and GTPO on LIBERO-Object with one-demonstration SFT initialization using seeds 1234, 5678, and 12345. Shuffled-GTPO permutes weights among eligible chunks within each failed trajectory while keeping all other settings unchanged.

MethodAUC-SR ↑Early SR ↑TTS@95 ↓
GRPO79.56 ± 1.7374.45 ± 0.6270 / >76 / >76
Shuffled-GTPO79.40 ± 1.9574.84 ± 2.2873 / >76 / >76
GTPO85.22 ± 0.8078.33 ± 0.1452 / 52 / 71

Metrics use the common interval of steps 0–76. AUC-SR and Early SR are mean ± sample standard deviation; Early SR is evaluated at step 19. TTS values list seeds 1234/5678/12345 separately, and >76>76 denotes that the threshold was not reached within this interval.

GTPO achieves higher AUC-SR and Early SR and reaches 95% success sooner on each seed. Shuffled-GTPO performs similarly to GRPO overall. This supports the importance of the correspondence between weights and action chunks, beyond merely introducing nonuniform weights. Mean-one normalization alone does not guarantee identical update behavior.

The original training experiments each used one independent run per setting. The additional comparisons show consistent performance across the evaluated seeds on LIBERO-Object. Results are provided in Section 4.3.2 and Appendix B.6.

[W2 & Q2: Trajectory Length, Normalization, and Weight Distribution]

R2: We agree that the original weighting rule has an explicit length dependence. Writing the transport-weighted row-mean cost as cˉi,t\bar c_{i,t}, the softmax logits are

cˉi,tLiτ.\frac{\bar c_{i,t}}{L_i\tau}.

For the same cost spread and temperature, shorter failure prefixes therefore produce sharper relative weights. Row normalization would remove this explicit 1/Li1/L_i factor. Per-trajectory range normalization would additionally remove cost-range differences, but could amplify small fluctuations when that range is close to zero.

Appendix B.8 and Figure 11 report per-iteration weight quantiles from seed 1234:

IterationP5P50P95
00.2491.0341.849
390.6981.0381.240
850.9371.0041.057

These statistics cover failed-trajectory chunks receiving OT credit, excluding successful trajectories, final chunks fixed at 1, and padding. They show that weights become closer to 1 during training.

[W3 & Q3: Failure Localization, Temporal Masking, and Unequal Lengths]

R3: Our current simulation setup provides task outcomes but does not directly label when a failure-inducing action occurs. Establishing such labels requires manual video inspection and involves subjective judgment. Figure 7 presents an actual rollout examined this way. The improvements over GRPO and shuffled-GTPO support the practical utility of these weights for learning.

The default temporal window is indeed broad. It provides a loose constraint, while visual costs and OT determine alignment within the window. In the bandwidth ablation, k=30k=30 achieves the highest AUC-SR (87.80) and shortest TTS@95 (52) among the tested settings, outperforming both narrower windows and k=50k=50 on these metrics (Table 6). This supports its effectiveness in the evaluated setting.

Balanced OT accommodates unequal sequence lengths, and our mask uses normalized time coordinates in that case. However, this does not eliminate forced matching when one trajectory lacks phases present in the other, including truncated failure prefixes. Unbalanced or partial OT are possible directions for further investigation.

AuthorsOpenReview ↗

Response to Reviewer x2jg

[Q4: Fixed τ=1\tau=1 Compared with GRPO]

R4: Our original comparison used different training seeds, which was an experimental oversight. We have added a same-seed comparison using seed 12345.

Fixed τ=1\tau=1 and GRPO show similar learning curves and nearly identical AUC-SR over steps 0–167: 86.98 versus 86.65, respectively. Their TTS@95 values are 133 and 110. Because TTS records the first threshold crossing, this difference can coexist with comparable overall learning curves. These results are consistent with near-uniform weighting behaving similarly to GRPO in this setting. The comparison is reported in Appendix B.7, Figure 10, and Table 12.

[Q5: Pointwise-Score Scaling]

R5: Yes. In the original Table 4 experiments, we divided each pointwise cosine distance by LiL_i, the number of aligned chunks in the failed rollout, before applying the softmax. This matches the 1/Li1/L_i row mass in the OT score.

We omitted this factor from the original description and have now made it explicit in Section 4.3.1 and Appendix B.3. This corrects the description of the experiments already performed.

[W5: Additional Credit-Assignment Baselines]

R6: GraphGPO offers a valuable direction for credit assignment, but applying it to continuous visual robot control requires adapting and validating its state-merging mechanism. Current resource constraints prevent us from completing this work, so we consider it and the first-divergence control directions for future research.

Regarding PPO/GAE, Table 1 already includes the published πRL results from RLinf: π0.5 with Flow-Noise, trained using PPO and GAE, achieves 98.3% average success on LIBERO. This is a published reference comparison retaining its original backbone and initialization.

Using TemporalOT alignment costs as per-step rewards requires design and validation within our training framework. These experiments remain incomplete due to limited computational resources; we will update the results as soon as they are completed.

Thank you for the detailed response and the new experiments. The shuffled-weight control with three seeds addresses my main concern. However, this control covers only LIBERO-Object, and it would be good to confirm it on Spatial or Goal, where training is longer.

One remaining question concerns the weight quantiles in R2. Their spread seems larger than Eqs. (13)–(14) allow for trajectories of 39 chunks. Could you check whether this comes from shorter failed rollouts, incomplete convergence, or something else?

The paper still lacks comparisons with other credit assignment baselines such as GraphGPO under matched conditions. A simpler control that concentrates credit at the first divergence from the teacher could be built from the existing OT alignment.

Failure localization remains the main open issue. Since LIBERO provides object poses and grasp checks, an analysis of where the weight mass falls relative to failure events seems feasible.

I also suggest stating in the paper that the weights become nearly uniform after the temperature annealing.

Reviewer_iaFzOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GTPO addresses sparse temporal credit assignment in GRPO-based reinforcement learning for vision-language-action policies. Each failed rollout is paired with a successful rollout from the same group using temporally masked optimal transport over visual features. Alignment costs determine normalized chunk-level weights that redistribute the failed rollout’s negative advantage, while successful rollouts retain standard GRPO updates. Experiments on LIBERO, RoboTwin, and five real-robot tasks demonstrate consistent improvements over matched GRPO baselines in success rate and training efficiency. Additional ablations investigate alignment, teacher selection, and weighting parameters.

Critical strengths
  1. A well-motivated approach to sparse credit assignment. GTPO addresses a concrete limitation of trajectory-level GRPO: uniformly assigning negative feedback across failed attempts despite differences in individual actions’ quality. Using temporal alignment to redistribute this feedback is a reasonable and focused algorithmic contribution.

  2. Practical methodology with clean information requirements. The method uses successful rollouts already available within the training group and visual features supplied by the policy. It requires neither a separately trained reward model nor additional deployment-time machinery. These modest additional requirements make it an appealing extension to existing GRPO training pipelines.

  3. Consistent gains under meaningful matched comparisons. GTPO improves matched GRPO performance across both demonstration settings on LIBERO, all four evaluated RoboTwin tasks, and all five real-robot tasks. LIBERO average success increases from 94.38% to 98.65% with one demonstration per task for initialization, while real-robot gains range from 10 to 35 percentage points. Matched initialization and training protocols strengthen these comparisons, and the training-efficiency results and component ablations provide evidence beyond final success rates.

Critical weaknesses
  1. The role of meaningful temporal alignment could be isolated more directly. The pointwise-matching ablation supports the proposed alignment procedure, but does not fully distinguish meaningful temporal allocation from the effects of nonuniform update weighting. Normalizing weights to average one does not preserve the gradient or clipping behavior. A shuffled-weight control would help establish whether assigning larger penalties to the identified chunks is essential. This limits the mechanistic interpretation rather than invalidating the demonstrated performance improvements.

  2. Uncertainty reporting could be stronger. The ablation curves do not quantify variability across independent training seeds, and real-robot evaluation uses 20 trials per task and checkpoint. Repeated runs or uncertainty estimates would improve confidence in smaller performance differences and comparisons among design choices, although the consistent improvements across tasks are encouraging.

Questions for authors
  1. Experimental request: How does GTPO perform if its computed weights are randomly permuted across eligible chunks within each failed rollout? Preserving the weight distribution while disrupting temporal correspondence would help distinguish meaningful alignment from the benefits of nonuniform weighting.

  2. Clarification: How many independent training runs support the main results and ablations? If repeated runs are available, how consistent are the improvements in success rate and training efficiency?

2 comments
AuthorsOpenReview ↗

Response to Reviewer iaFz

Thank you for highlighting the need to distinguish temporal correspondence from generic nonuniform weighting and to strengthen uncertainty reporting.

[W1 & Q1: Shuffled-Weight Control]

R1: We agree that normalizing weights to mean one does not preserve the gradient or clipping behavior. We therefore compared GRPO, shuffled-GTPO, and GTPO on LIBERO-Object with one-demonstration SFT initialization using seeds 1234, 5678, and 12345.

Shuffled-GTPO randomly permutes the computed weights among eligible chunks within each failed rollout. This preserves their values and sum while disrupting their temporal correspondence; all other settings remain unchanged.

MethodAUC-SR ↑Early SR ↑
GRPO79.56 ± 1.7374.45 ± 0.62
Shuffled-GTPO79.40 ± 1.9574.84 ± 2.28
GTPO85.22 ± 0.8078.33 ± 0.14

Values are mean ± sample standard deviation across seeds over the common training interval of steps 0–76. Early SR is evaluated at step 19. GTPO also reaches 95% success sooner on every seed.

Shuffled-GTPO performs similarly to GRPO overall, whereas GTPO consistently improves these learning-efficiency metrics. This supports the value of assigning weights to their corresponding action chunks. The results are included in Section 4.3.2, Figure 5, Table 5, and Appendix B.6.

[W2 & Q2: Independent Training Runs and Consistency]

R2: Each setting in the training experiments we conducted for the original submission used one independent run. The additional LIBERO-Object comparisons now evaluate all three methods across the seeds above and show consistent GTPO advantages in AUC-SR, Early SR, and TTS@95.

These results strengthen the evidence for this setting. The π0.5 simulation and RLT real-robot comparisons are also underway, but limited computational and robot resources constrain their completion. If additional computational resources become available, we will extend the multi-seed experiments to other LIBERO suites.

[W2: Uncertainty in Real-Robot Evaluation]

R3: We have expanded the real-robot evaluation to 60 trials per task and checkpoint: the original 20 trials plus two additional batches of 20. All three batches evaluate the same trained checkpoint. We pool the success counts and report 95% Wilson confidence intervals, which quantify evaluation uncertainty for the fixed policy. The complete learning curves are provided in Appendix C.7, Figure 12.

Repeating the full real-robot training pipeline is costly, and no additional compute resources were available for independent training repetitions.

Very nice clarification, maintaining my assessment!

Reviewer_W1iBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Reinforcement Learning algorithm to improve robotic foundation models, especially Vision-Language-Action models. The authors propose GTPO, which uses trajectory comparison to refine otherwise uniform trajectory-level advantages. Each failed rollout is paired with its nearest successful in-group teacher using the policy's visual features. Temporally masked optimal transport aligns their trajectories and produces normalized action-chunk weights that redistribute the original GRPO advantage while retaining terminal rewards. The teachers evolve with the policy, requiring no additional expert reference trajectories or separately trained reward model for teaching. Extensive experiments on LIBERO, RoboTwin, and real-world tasks demonstrate the proposed method's superior performance.

Critical strengths
  1. This paper studies how to fine-tune robot foundation models such as VLAs with online reinforcement learning, which is an important and timely research topic;

  2. The paper conducts comprehensive experiments to evaluate the effectiveness of their proposed method;

  3. The authors further evaluate their method on challenging real-robot experiments.

Critical weaknesses
  1. The authors do not make it very clear about what base VLA models they use in RL methods in main results. I assume they all use OpenVLA-OFT as base VLA models (correct me if this is wrong). OpenVLA directly encodes robot actions as tokens in vocabulary, which is different from modern VLA models that employ flow matching action expert to generate continuous robot actions. It is important to discuss whether the proposed method works with modern VLA models such as pi0.5. I do find the authors use pi0.5 in their real robot experiments. Could you please illustrate why the simulation experiments only use OpenVLA and the real experiments use pi0.5, and demonstrate with simulation experiments that the proposed method works well with flow-matching based VLA models ?

  2. The authors lack discussion or comparison with other representative online RL methods, such as RL Tokens, DSRL, and Policy Decorator. I am curious to know the performance comparison between GTPO and the methods I mention above.

Questions for authors

See the weakness points I list.

7 comments
AuthorsOpenReview ↗

Response to Reviewer W1iB

Thank you for your constructive comments on the backbone choices and comparisons with other RL methods.

[W1: Backbone Choices and Applicability to Flow-Matching Policies]

R1: All GRPO and GTPO simulation experiments in the original submission used OpenVLA-OFT. We deliberately selected π0.5 for the real-robot experiments to evaluate GTPO with a flow-matching action expert and examine its applicability across different VLA architectures.

We have clarified the backbone choices in the revised experimental descriptions and table captions. A π0.5 simulation comparison is now running, but completing it requires additional time under our available computational budget.

[W2: Comparisons with RLT, DSRL, and Policy Decorator]

R2: We are also keen to understand how GTPO compares with strong recent approaches such as RLT, and are conducting real-robot comparisons with RLT. We will update the results as soon as all experiments are completed.

Our available robot time does not currently permit additional comparisons with DSRL and Policy Decorator.

AuthorsOpenReview ↗

Response to Reviewer W1iB

Thank you for suggesting this comparison. We have added real-robot experiments with RLT using the same three-round training and evaluation protocol, with 60 evaluation trials per task and checkpoint. Each entry below reports GTPO / RLT success rates (%).

TaskRound 1Round 2Round 3
Stacking96.67 / 96.6795.00 / 95.0098.33 / 100.00
Battery insertion73.33 / 75.0088.33 / 88.3396.67 / 96.67
Soldering75.00 / 76.6791.67 / 90.0093.33 / 91.67
Screw tightening46.67 / 50.0060.00 / 63.3376.67 / 80.00
Cap removal81.67 / 83.3395.00 / 90.0093.33 / 96.67
Average74.67 / 76.3386.00 / 85.3391.67 / 93.00

RLT performs strongly, and the two methods achieve similar success rates across training rounds. Their optimization approaches differ: RLT freezes the VLA and trains a lightweight off-policy actor–critic through an RL-token interface, whereas GTPO directly optimizes the VLA policy through on-policy, critic-free group-relative updates. GTPO requires neither a learned critic nor an additional RL-token extractor or control head, and its temporal credit assignment mechanism has been evaluated with both OpenVLA-OFT and the flow-matching π0.5 policy. These results demonstrate competitive real-robot performance with our critic-free approach.

I appreciate the authors’ efforts in providing additional results. My remaining concern is that off-policy RL methods generally achieve higher sample efficiency than on-policy methods, as your comparison with RLT also indicates, and real-world robotic RL typically relies on off-policy algorithms. What is the motivation for developing an on-policy approach? You note that GTPO is critic-free, which I am not sure whether is an important advantage or not, Why do you believe robotic RL requires a critic-free method at the current model scale?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We thank the Area Chair and all reviewers for their careful evaluations and constructive suggestions. We appreciate Reviewer W1iB’s recognition of the importance of VLA reinforcement learning and our real-robot evaluation, Reviewer iaFz’s positive assessment of our motivation and matched comparisons, and Reviewer x2jg’s recognition of the improvements in success rate and training efficiency.

Following the feedback, we have strengthened the experimental evidence and clarified the method and experimental descriptions:

  • Temporal correspondence and consistency across seeds (iaFz, x2jg). We added comparisons between GRPO, shuffled-GTPO, and GTPO on LIBERO-Object using different training seeds. GTPO consistently achieves higher AUC-SR and Early SR and reaches 95% success sooner. Shuffled-GTPO performs similarly to GRPO overall, supporting the value of assigning weights to their corresponding action chunks.
  • Pointwise-score scaling and temperature comparison (x2jg). We clarified that the original pointwise experiments already divided the costs by the aligned trajectory length before softmax. We also added a same-seed comparison showing that fixed τ=1\tau=1 and GRPO have similar learning curves and nearly identical AUC-SR.
  • Weight concentration and interpretation (x2jg). We added empirical weight quantiles and an explanation of trajectory-length dependence, together with a visualization and experimental evidence illustrating the role of visual alignment in policy learning.
  • Backbones and additional baselines (W1iB). We clarified the use of OpenVLA-OFT in the original simulation experiments and π0.5 in the real-robot experiments. Additional π0.5 simulation and RLT real-robot comparisons are underway; limited computational and robot resources mean they require more time to complete. We will update the RLT results as soon as all experiments are completed.
  • Real-robot evaluation uncertainty (iaFz). Each task and checkpoint now has 60 evaluation trials in three batches of 20. We report 95% Wilson confidence intervals for these fixed checkpoints.

We also standardized AUC-SR computation to match the original Table 4, resulting in minor numerical updates to the ablation results without changing the underlying training records.

The revised manuscript includes the shuffled-weight results in Section 4.3.2, the pointwise clarification in Section 4.3.1 and Appendix B.3, and the additional analyses in Appendices B.6–B.8. Revisions are highlighted in blue. We address the individual concerns below.

AuthorsOpenReview ↗

Final Summary of Discussion and Updates (To Area Chair and Reviewers)

Dear Area Chair and Reviewers,

As the discussion period wraps up, we want to sincerely thank Reviewers iaFz, x2jg, and W1iB for such a constructive and engaging dialogue. Your feedback pushed us to put GTPO through stricter controls, stronger baselines, and an additional VLA architecture—and we are glad to report that every new experiment further reinforces the core thesis of the paper.

Below is a brief summary of how the main points raised during the discussion have been addressed and incorporated into our revised manuscript:

1. Temporal Correspondence—Not Just Nonuniform Weighting—Drives the Gain (iaFz, x2jg)

To verify that GTPO's improvements genuinely come from aligning weights to the right action chunks, we added a 3-seed shuffled-weight control on LIBERO-Object and extended it to the longer-horizon LIBERO-Spatial suite:

  • LIBERO-Object: GTPO achieves 85.22±0.8085.22 \pm 0.80 AUC-SR (vs. 79.40±1.9579.40 \pm 1.95 for Shuffled-GTPO and 79.56±1.7379.56 \pm 1.73 for GRPO) and hits 95% success faster on every seed.
  • LIBERO-Spatial: The gap widens further—GTPO reaches 90.96 AUC-SR (crossing 95% at step 110), compared with 85.42 for Shuffled-GTPO (step 166) and 84.87 for GRPO (remaining below 95% through step 206).

We are pleased that Reviewer iaFz found our clarifications convincing and maintained their positive assessment, and that Reviewer x2jg noted the multi-seed shuffled control addressed their primary concern.

2. Clear Advantages Over Alternative Credit-Assignment Baselines & Clarified Mechanics (x2jg)

Following Reviewer x2jg's follow-up suggestions, we added two matched credit-assignment baselines on LIBERO-Object: an OT-reward GRPO control and a visual-control adaptation of GraphGPO:

  • GTPO (85.97 AUC-SR, TTS@95 52) outperforms both OT-reward GRPO (78.41, TTS@95 >111> 111) and GraphGPO (80.08, TTS@95 >76> 76), confirming that redistributing trajectory advantages via temporal OT alignment is substantially more effective than collapsing OT costs into scalar rewards or merging continuous visual states into discrete graphs.
  • We also reconciled the Figure 11 temperature schedule (showing the 0.01→10.01 \to 1 and 0.05→10.05 \to 1 runs remain consistent within 1.04 AUC-SR points), confirmed same-seed parity between fixed τ=1\tau=1 and GRPO (86.98 vs. 86.65), and explicitly clarified the 1/Li1/L_i scaling and near-uniform post-annealing weight behavior in the paper.

3. Cross-Architecture Generality and Strong Critic-Free Real-Robot Performance (W1iB, iaFz)

  • Real-Robot Robustness & RLT Comparison: We expanded real-robot testing to 60 trials per task and checkpoint with 95% Wilson confidence intervals and added a direct comparison against the strong off-policy actor–critic baseline RLT. Across five manipulation tasks over three rounds, GTPO improves from 74.67%→86.00%→91.67%74.67\% \to 86.00\% \to 91.67\%, closely tracking RLT (76.33%→85.33%→93.00%76.33\% \to 85.33\% \to 93.00\%) while staying completely on-policy and critic-free—avoiding any learned critic, RL-token extractor, auxiliary control head, or critic cold-start instability under sparse real-world rewards. We also appreciated the insightful exchange with Reviewer W1iB on extending our dense credit-assignment mechanism to off-policy paradigms such as AWR and AWBC.
  • Flow-Matching VLA Validation in Simulation: We have now completed the additional π0.5\pi_{0.5} simulation experiments across LIBERO-Object, Spatial, and Goal. Even starting from high-performing SFT checkpoints where RL headroom is naturally tight, GTPO consistently outperforms GRPO across all three suites (+1.43 percentage points average AUC-SR: 96.30 vs. 95.60 on Object; 93.92 vs. 92.56 on Spatial; 90.00 vs. 87.77 on Goal), alongside stronger early success rates and faster convergence. We will include the remaining LIBERO-Long π0.5\pi_{0.5} results in the camera-ready version.

Overall, this discussion has made the paper markedly stronger—solidifying GTPO's consistency across seeds, its edge over competing credit-assignment baselines, its generality across both autoregressive (OpenVLA-OFT) and flow-matching (π0.5\pi_{0.5}) VLA backbones, and its practical value on real robots. Thank you again to the Area Chair and all reviewers for your time, care, and thoughtful feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.