- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Existing work has used trajectory-generating policies as an interface (i.e., 2D coordinates) to a diffusion action head in robotics control. TCoT-RL builds off that work by introducing twin GRPO objectives to explore in both the trajectory space and control space at the same time. A potential-based task-progress reward is added measured using known object positions, along with a stage-aware endpoint reward that measures how close the final position is to the known required final location for the current subtask. This reward shaping is used for both the trajectory generation and the action generation, but the actual environment rewards from the task are only used for the action generation. There is one action generation per trajectory generation. Evaluations on LIBERO and LIBERO-Plus show state-of-the-art results when averaged over all test domains, both when using extra demonstrations for prior SFT and using restricted demonstrations for prior SFT, in addition to providing several ablations and state-coverage analysis.
Critical strengths
The greatest strengths of this paper are its simplicity, breadth of experiments, and state-of-the-art results. Table 1 shows a comparison to many methods, and TCoT-RL is the strongest. The method itself is fairly straightforward, i.e., perform RL on both trajectories and on environment actions. The ablations and analysis are also thoughtful, with helpful commentary.
Critical weaknesses
The most critical weakness of the paper are the lack of multiple seeds (as far as I can tell) for the main results, a lack of simple alternative exploration techniques common in RL, and idiosyncratic sensitivity to the shaped rewards and environment rewards at different levels. Specifically, Appendix I shows that confidence intervals can span over 4% points, which is insufficient to significantly distinguish TCoT-RL from other methods. Additionally, there are simple exploration baselines in actions space (e.g., RND, entropy bonus, Diversity is All You Need) that leave me wondering whether there are easier fixes here. Finally, the fact that environment rewards cant be used to train the trajectories seems to place a heavy burden on manually engineering rewards. That said, these may be addressable.
Questions for authors
- Do you have any idea why the shaped rewards can be used in conjunction with task rewards for the action-level RL, but the task rewards cannot be used with the trajectory-level RL?
- How does exploration in trajectory space compare to simple exploration baselines in actions space (e.g., RND, entropy bonus, Diversity is All You Need (DIAYN))?
- Any idea how a trajectory-only RL ablation would compare?
- Is an underline in Table 1 supposed to indicate second best? Because TCoT-RL does not have the second best Overall SR. Is this an error?
- In figure 1 (c) it looks like TCoT-RL cannot achieve as high C40 at the action level. If you consider the Pareto Frontier of this plot, there are only two points remain, and the existing VLA-RL point (blue) looks more favorable. Am I missing something here?
- Section 4.4 seems to draw heavy conclusions from a highly smoothed graph, whereas the underlying data seems to be highly ambiguous, and potentially even support the conclusion that Fixed-10 waypoints can achieve higher success rates. Is that right?
- Which baselines were re-trained and which are results reported from the original papers? For methods that are re-trained, was equal compute given to baseline tuning?
Minor:
- “During trajectory-supervised fine-tuning, the trajectory policy is trained autoregressively to predict this sequence”: Does this mean you don’t do teacher forcing?
- Why is the end reward clipped to [0,1]? Shouldn’t the min be 0 anyway?
- Why is the milestone reward maxed with zero? Does this not break potential-based rewards and mean the agent can keep collecting rewards by accomplishing the same milestone over and over aging?
- Can you explain the baselines so that readers have sufficient context to understand the comparison without needing to search externally for the papers?
- The point of Figure 3 is unclear to me. Some more text in the caption could help.
- What is C40 in figure 1 (c) and 20^2, 40^2, first 5, all in figure 1 (d)? Explaining this with the figure and later in the analysis section could be helpful.
- For future work, I wonder how this compares to RL on the reasoning, as in Adapting Generalist Robot Policies with Semantic Reinforcement Learning (Bhatia et al.)
8 comments
Response to ZFHw (Part 2/2)
Q4 (underline in Table 1). Underlines mark the best result among methods without LIBERO-Plus SFT data (caption). In that group, TCoT-RL (86.8) is best overall, ahead of π0.5 (85.7). The revised table separates the two settings visually.
Q5 (Fig. 1c). Each point is one matched reset, not a method, so a Pareto frontier does not apply. Orange points are resets where trajectory-level exploration covers more cells: 36 of 50, with a mean gain of +10.5 cells. Point positions in the scatter are approximate; the statistics are computed from the raw per-reset values (Figs. 9–10). The caption now explains this.
Q6 (Sec. 4.4). We agree and have weakened the claim: adaptive allocation performs at least as well as the best fixed budget, without choosing the waypoint count by hand. The online curves come from exploratory training rollouts and fluctuate from update to update, so we now rely on the evaluation on 128 fixed episodes (81.3% vs. 78.9% for Fixed-2 and 75.8% for Fixed-10).
Q7 (retrained vs. reported baselines). In Tables 1 and 2, every result except TCoT-RL is taken from the original paper or the public leaderboard (π0.5 in Table 1 is the official result); we did not rerun these methods, so no tuning on our side is involved. All rows of Table 3 are retrained by us in the same codebase with the same number of training steps, and the matched baselines in App. N are likewise our own runs under the same seeds, resets, and interaction budget. This is now stated in Sec. 4.1 and App. B.
Minor points.
- Teacher forcing: yes, standard teacher-forced next-token cross-entropy; "autoregressively" referred to generation. Fixed.
- Clip in r_end: the lower bound is redundant; we now write min(·, 1).
- Milestone term: each milestone is rewarded at most once per episode (the stage is tracked as a running maximum), so re-reaching a stage after dropping the object earns nothing; the notation is fixed. You are right that the reward is not strictly potential-based. We removed that claim and now call it dense progress shaping, with the reasons given in App. C.3.
- Baseline sources: stated in Sec. 4.1 and App. B (see Q7).
- Fig. 3 caption: it now identifies the rows (top: action-only RL; bottom: TCoT-RL); the row labels in the App. K caption were swapped and are corrected.
- C40, 20²/40², and first 5 / all in Fig. 1: defined in the caption and in Sec. 4.3.
- Bhatia et al.: added.
Thank you for the detailed follow up. This does clarify some ambiguity.
Regarding Figure 1 (c), if I understand correctly, the blue and orange dots (which compare two different methods) both have similar y-values, and therefore similar trajectory coverage. However, the blue dots lie to the right of y=x, so they have more action coverage as well, at a given level of trajectory coverage. Is this not advantage to the baseline (blue)?
Reviewer ZFHw