- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GTPO, RL approach for vision-language action policies that improves trajectory-level GRPO advantages using successful rollouts as teachers. For each failed trajectory method selects the closest successful rollout, computes a temporally constrained optimal transport alignment over visual features, and uses the resulting costs to redistribute negative advantages across actions.
Critical strengths
- The paper addresses an important problem: extracting informative temporal supervision from sparse terminal rewards without training an additional model.
- The method achieves improvements over GRPO on LIBERO and RoboTwin, with the largest gains from the one-demonstration warm start.
- The method improves training efficiency (AUC-SR, Early SR).
- The evaluation includes real-world robotic manipulation tasks.
- The appendix is detailed, and the limitations of visual comparison are stated.
Critical weaknesses
W1. No comparison with randomly shuffled GTPO weights. This comparison is necessary to distinguish meaningful temporal credit assignment from generic non-uniform advantage reweighting. A trajectory permutation keeps the mean weight at 1, so it matches the GRPO gradient in expectation while adding the same variance.
W2. Dependence of weight concentration on trajectory length. Since the transport plan has uniform row marginals , , and the weight ratio is bounded by . Longer trajectories therefore have a higher effective softmax temperature and more uniform credit weights, which reduces the ability to distinguish important actions. This matters most in the real-world setting, where failed prefixes are truncated at takeover and . In simulation (), the weights differ by at most about 5% once reaches 1. App. B.3 acknowledges the scale issue but does not resolve it.
W3. Limited validation of action failure localization. Visual dissimilarity does not necessarily identify the failure-inducing action. An incorrect action may cause visual divergence at later stages, leading GTPO to penalize the following actions rather than the failure-inducing one; in Fig. 6 the weights peak several chunks after the annotated failed grasp. The authors acknowledge this limitation, but localization is not quantified, although simulation provides ground-truth failure events. The temporal mask is also nearly vacuous at the defaults ( against ), and balanced OT may distort the alignment when trajectory lengths differ (an early-terminating teacher versus a failure that runs to the horizon, or a truncated real-world prefix). Unbalanced or partial OT, which does not force every chunk of the failure to be matched, may be more appropriate.
W4. Statistical support. The results appear to come from single training runs.
W5. Baselines GRPO is the only baseline trained under matched conditions. The closest related approach is graph-based credit assignment (GraphGPO, Cheng et al., ICML 2026), which merges shared prefixes of group rollouts into a state-transition graph and assigns credit at the branching points where trajectories diverge, i.e., where the failure-inducing action should be. Exact state merging does not apply to continuous visual observations, but an approximate variant (merging states within a feature-distance threshold, or concentrating negative credit at the first divergence from the teacher) would be a natural baseline and would directly test W3. TemporalOT used as a per-step reward or PPO/GAE would also clarify the contribution.
Questions for authors
- How does GTPO compare to randomly permuted weights, with three seeds for GTPO, permuted GTPO, and GRPO?
- Have the authors considered a row-normalized alignment cost, , or a per-trajectory range normalization? How would this affect performance on trajectories of different lengths, e.g., the truncated real-world prefixes? What is the empirical distribution of over training?
- How strongly do the OT-derived credit weights correlate with the actions actually responsible for failures, e.g., relative to simulator failure events?
- Why constant (near-uniform weights) underperforms GRPO on TTS@95?
- In Table 4, the pointwise score lacks the transport mass contained in the OT score, so at the same it is about times larger and gives much sharper weights (as noted in App. B.3). Was it rescaled by ? Otherwise the comparison conflates the alignment method with weight concentration.
5 comments
Response to Reviewer x2jg
[Q4: Fixed Compared with GRPO]
R4: Our original comparison used different training seeds, which was an experimental oversight. We have added a same-seed comparison using seed 12345.
Fixed and GRPO show similar learning curves and nearly identical AUC-SR over steps 0–167: 86.98 versus 86.65, respectively. Their TTS@95 values are 133 and 110. Because TTS records the first threshold crossing, this difference can coexist with comparable overall learning curves. These results are consistent with near-uniform weighting behaving similarly to GRPO in this setting. The comparison is reported in Appendix B.7, Figure 10, and Table 12.
[Q5: Pointwise-Score Scaling]
R5: Yes. In the original Table 4 experiments, we divided each pointwise cosine distance by , the number of aligned chunks in the failed rollout, before applying the softmax. This matches the row mass in the OT score.
We omitted this factor from the original description and have now made it explicit in Section 4.3.1 and Appendix B.3. This corrects the description of the experiments already performed.
[W5: Additional Credit-Assignment Baselines]
R6: GraphGPO offers a valuable direction for credit assignment, but applying it to continuous visual robot control requires adapting and validating its state-merging mechanism. Current resource constraints prevent us from completing this work, so we consider it and the first-divergence control directions for future research.
Regarding PPO/GAE, Table 1 already includes the published πRL results from RLinf: π0.5 with Flow-Noise, trained using PPO and GAE, achieves 98.3% average success on LIBERO. This is a published reference comparison retaining its original backbone and initialization.
Using TemporalOT alignment costs as per-step rewards requires design and validation within our training framework. These experiments remain incomplete due to limited computational resources; we will update the results as soon as they are completed.
Thank you for the detailed response and the new experiments. The shuffled-weight control with three seeds addresses my main concern. However, this control covers only LIBERO-Object, and it would be good to confirm it on Spatial or Goal, where training is longer.
One remaining question concerns the weight quantiles in R2. Their spread seems larger than Eqs. (13)–(14) allow for trajectories of 39 chunks. Could you check whether this comes from shorter failed rollouts, incomplete convergence, or something else?
The paper still lacks comparisons with other credit assignment baselines such as GraphGPO under matched conditions. A simpler control that concentrates credit at the first divergence from the teacher could be built from the existing OT alignment.
Failure localization remains the main open issue. Since LIBERO provides object poses and grasp checks, an analysis of where the weight mass falls relative to failure events seems feasible.
I also suggest stating in the paper that the weights become nearly uniform after the temperature annealing.
Response to Reviewer x2jg
Thank you for the detailed analysis of the weighting mechanism and the suggested controls. We have added the shuffled-weight comparison, a same-seed temperature comparison, and empirical weight statistics, and clarified the pointwise-score scaling.
[W1, W4 & Q1: Shuffled Weights and Independent Training Runs]
R1: We compared GRPO, shuffled-GTPO, and GTPO on LIBERO-Object with one-demonstration SFT initialization using seeds 1234, 5678, and 12345. Shuffled-GTPO permutes weights among eligible chunks within each failed trajectory while keeping all other settings unchanged.
Metrics use the common interval of steps 0–76. AUC-SR and Early SR are mean ± sample standard deviation; Early SR is evaluated at step 19. TTS values list seeds 1234/5678/12345 separately, and >76 denotes that the threshold was not reached within this interval.
GTPO achieves higher AUC-SR and Early SR and reaches 95% success sooner on each seed. Shuffled-GTPO performs similarly to GRPO overall. This supports the importance of the correspondence between weights and action chunks, beyond merely introducing nonuniform weights. Mean-one normalization alone does not guarantee identical update behavior.
The original training experiments each used one independent run per setting. The additional comparisons show consistent performance across the evaluated seeds on LIBERO-Object. Results are provided in Section 4.3.2 and Appendix B.6.
[W2 & Q2: Trajectory Length, Normalization, and Weight Distribution]
R2: We agree that the original weighting rule has an explicit length dependence. Writing the transport-weighted row-mean cost as cˉi,t, the softmax logits are
Liτcˉi,t.
For the same cost spread and temperature, shorter failure prefixes therefore produce sharper relative weights. Row normalization would remove this explicit 1/Li factor. Per-trajectory range normalization would additionally remove cost-range differences, but could amplify small fluctuations when that range is close to zero.
Appendix B.8 and Figure 11 report per-iteration weight quantiles from seed 1234:
These statistics cover failed-trajectory chunks receiving OT credit, excluding successful trajectories, final chunks fixed at 1, and padding. They show that weights become closer to 1 during training.
[W3 & Q3: Failure Localization, Temporal Masking, and Unequal Lengths]
R3: Our current simulation setup provides task outcomes but does not directly label when a failure-inducing action occurs. Establishing such labels requires manual video inspection and involves subjective judgment. Figure 7 presents an actual rollout examined this way. The improvements over GRPO and shuffled-GTPO support the practical utility of these weights for learning.
The default temporal window is indeed broad. It provides a loose constraint, while visual costs and OT determine alignment within the window. In the bandwidth ablation, k=30 achieves the highest AUC-SR (87.80) and shortest TTS@95 (52) among the tested settings, outperforming both narrower windows and k=50 on these metrics (Table 6). This supports its effectiveness in the evaluated setting.
Balanced OT accommodates unequal sequence lengths, and our mask uses normalized time coordinates in that case. However, this does not eliminate forced matching when one trajectory lacks phases present in the other, including truncated failure prefixes. Unbalanced or partial OT are possible directions for further investigation.