acceptodds
Under review as a conference paper at ICLR 2027

Thinking in Trajectories: Structured Exploration for Vision-Language-Action Reinforcement Learning

Abstract

Reinforcement-learning post-training has recently improved vision-language-action (VLA) models beyond supervised imitation, yet most existing methods explore by sampling low-level actions or action chunks. In a high-dimensional continuous action space, such perturbations mostly yield local control variations: changing the high-level intent of a long-horizon behavior requires coordinated deviations over many steps, which random action noise discovers only inefficiently. We propose TCoT-RL, which formulates trajectory reasoning as a directly optimizable exploration policy for VLA post-training: a trajectory policy autoregressively samples a variable-length sequence of end-effector waypoints representing a high- level motion plan, while a flow-based action policy generates continuous controls conditioned on the sampled trajectory. This hierarchical factorization enables structured exploration over alternative behavioral intents rather than only perturbing low-level actions. To directly optimize both levels of the hierarchy, TCoT-RL performs joint trajectory–action reinforcement learning with level-specific credit assignment. We introduce dense trajectory feedback based on task progress and stage-aware endpoint grounding, and optimize the trajectory and action policies with separate group-relative advantages and clipped GRPO objectives. This allows trajectory reasoning to evolve from a supervised intermediate representation into an active exploration policy through online interaction. Experiments show that TCoT-RL consistently improves over supervised trajectory-conditioned policies and matched action-only RL baselines. On LIBERO-Plus, TCoT-RL achieves a state-of-the-art success rate of 89.6%, outperforming the strongest competing RL baseline by 1.8 percentage points. Paired rollout analysis further shows that trajectory-level exploration expands end-effector state coverage and visitation entropy and produces greater behavioral diversity. These results demonstrate that explicitly optimizing the distribution over high-level motion hypotheses provides an effective structured exploration mechanism for VLA reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Vision-Language-Action Models, Reinforcement Learning, Trajectory-Level Exploration, Embodied Reasoning, Online Post-Training, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZFHwOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing work has used trajectory-generating policies as an interface (i.e., 2D coordinates) to a diffusion action head in robotics control. TCoT-RL builds off that work by introducing twin GRPO objectives to explore in both the trajectory space and control space at the same time. A potential-based task-progress reward is added measured using known object positions, along with a stage-aware endpoint reward that measures how close the final position is to the known required final location for the current subtask. This reward shaping is used for both the trajectory generation and the action generation, but the actual environment rewards from the task are only used for the action generation. There is one action generation per trajectory generation. Evaluations on LIBERO and LIBERO-Plus show state-of-the-art results when averaged over all test domains, both when using extra demonstrations for prior SFT and using restricted demonstrations for prior SFT, in addition to providing several ablations and state-coverage analysis.

Critical strengths

The greatest strengths of this paper are its simplicity, breadth of experiments, and state-of-the-art results. Table 1 shows a comparison to many methods, and TCoT-RL is the strongest. The method itself is fairly straightforward, i.e., perform RL on both trajectories and on environment actions. The ablations and analysis are also thoughtful, with helpful commentary.

Critical weaknesses

The most critical weakness of the paper are the lack of multiple seeds (as far as I can tell) for the main results, a lack of simple alternative exploration techniques common in RL, and idiosyncratic sensitivity to the shaped rewards and environment rewards at different levels. Specifically, Appendix I shows that confidence intervals can span over 4% points, which is insufficient to significantly distinguish TCoT-RL from other methods. Additionally, there are simple exploration baselines in actions space (e.g., RND, entropy bonus, Diversity is All You Need) that leave me wondering whether there are easier fixes here. Finally, the fact that environment rewards cant be used to train the trajectories seems to place a heavy burden on manually engineering rewards. That said, these may be addressable.

Questions for authors
  1. Do you have any idea why the shaped rewards can be used in conjunction with task rewards for the action-level RL, but the task rewards cannot be used with the trajectory-level RL?
  2. How does exploration in trajectory space compare to simple exploration baselines in actions space (e.g., RND, entropy bonus, Diversity is All You Need (DIAYN))?
  3. Any idea how a trajectory-only RL ablation would compare?
  4. Is an underline in Table 1 supposed to indicate second best? Because TCoT-RL does not have the second best Overall SR. Is this an error?
  5. In figure 1 (c) it looks like TCoT-RL cannot achieve as high C40 at the action level. If you consider the Pareto Frontier of this plot, there are only two points remain, and the existing VLA-RL point (blue) looks more favorable. Am I missing something here?
  6. Section 4.4 seems to draw heavy conclusions from a highly smoothed graph, whereas the underlying data seems to be highly ambiguous, and potentially even support the conclusion that Fixed-10 waypoints can achieve higher success rates. Is that right?
  7. Which baselines were re-trained and which are results reported from the original papers? For methods that are re-trained, was equal compute given to baseline tuning?

Minor:

  1. “During trajectory-supervised fine-tuning, the trajectory policy is trained autoregressively to predict this sequence”: Does this mean you don’t do teacher forcing?
  2. Why is the end reward clipped to [0,1]? Shouldn’t the min be 0 anyway?
  3. Why is the milestone reward maxed with zero? Does this not break potential-based rewards and mean the agent can keep collecting rewards by accomplishing the same milestone over and over aging?
  4. Can you explain the baselines so that readers have sufficient context to understand the comparison without needing to search externally for the papers?
  5. The point of Figure 3 is unclear to me. Some more text in the caption could help.
  6. What is C40 in figure 1 (c) and 20^2, 40^2, first 5, all in figure 1 (d)? Explaining this with the figure and later in the analysis section could be helpful.
  7. For future work, I wonder how this compares to RL on the reasoning, as in Adapting Generalist Robot Policies with Semantic Reinforcement Learning (Bhatia et al.)
8 comments
AuthorsOpenReview ↗

Reviewer ZFHw

Thank you for the detailed review. G1–G2 address seeds and reward engineering.

W1 (multiple seeds; Appendix I intervals). See G1. The Wilson intervals in App. I reflect finite-evaluation uncertainty for each run separately. Because all methods share the same episodes, a paired comparison is more appropriate. TCoT-RL wins on every seed (+2.23 ± 0.49 pp over Action-only GRPO), with seed std of only 0.2–0.4 pp.

W2/Q2 (simple exploration baselines). For flow-based action experts the action entropy has no closed form; the flow-SDE noise scale is the practical analogue of an entropy bonus, as it directly sets how widely actions are sampled. We therefore retrained Action-only GRPO with this noise doubled during training (0.5 → 1.0), with the same seed and budget (24 updates), and evaluated all policies on the same 128 LIBERO-Plus episodes (new App. S):

  • Action-only GRPO, default noise: 109/128 (85.2%);
  • Action-only GRPO, doubled noise: 107/128 (83.6%);
  • TCoT-RL: 117/128 (91.4%).

Stronger action-level exploration does not improve success, so the simplest fix does not close the gap. (The 128-episode subset is used only for this comparison; for the size of the gain we rely on the three-seed, 1,024-episode results in G1, +2.23 ± 0.49 pp.) This is consistent with Sec. 4.3: more action noise mainly adds local control variation, whereas trajectory sampling changes the initial intent (+20.0% diversity over the first five actions).

Q1 (why not task reward for the trajectory policy). This is a design choice, not a constraint: the task reward can be added, but it does not help. Whether an episode succeeds depends mostly on how the remaining actions are executed, not on which trajectory was sampled for one chunk, so it tells the trajectory policy very little.

  • Measured (new App. P). From fixed simulator states, we sampled 8 trajectories and executed each 8 times. The choice of trajectory explains under 1% of the variance in episode success, versus 45% for the per-chunk trajectory reward we use.
  • Tested (new App. T). Adding the task reward to the trajectory objective gives 86.5%, versus 86.8% without it (protocol of Table 3).

Q3 (trajectory-only RL). We added it (new row in Table 3; App. T). With the action expert frozen at the SFT checkpoint and only the trajectory policy optimized, success is 86.1%, vs. 85.5% for TCoT-SFT, 85.8% for action-only RL, and 86.8% for joint training. Optimizing the trajectory policy alone thus already matches or slightly exceeds optimizing the action policy alone, which supports our claim that the trajectory distribution is the key object to optimize. Joint training remains best: the trajectory policy changes the intent, and the action policy learns to execute the new intents. These are single runs with differences within 0.7 pp, so we present them as descriptive rather than significant.

(Part 1/2; continued below.)

AuthorsOpenReview ↗

Response to ZFHw (Part 2/2)

Q4 (underline in Table 1). Underlines mark the best result among methods without LIBERO-Plus SFT data (caption). In that group, TCoT-RL (86.8) is best overall, ahead of π0.5 (85.7). The revised table separates the two settings visually.

Q5 (Fig. 1c). Each point is one matched reset, not a method, so a Pareto frontier does not apply. Orange points are resets where trajectory-level exploration covers more cells: 36 of 50, with a mean gain of +10.5 cells. Point positions in the scatter are approximate; the statistics are computed from the raw per-reset values (Figs. 9–10). The caption now explains this.

Q6 (Sec. 4.4). We agree and have weakened the claim: adaptive allocation performs at least as well as the best fixed budget, without choosing the waypoint count by hand. The online curves come from exploratory training rollouts and fluctuate from update to update, so we now rely on the evaluation on 128 fixed episodes (81.3% vs. 78.9% for Fixed-2 and 75.8% for Fixed-10).

Q7 (retrained vs. reported baselines). In Tables 1 and 2, every result except TCoT-RL is taken from the original paper or the public leaderboard (π0.5 in Table 1 is the official result); we did not rerun these methods, so no tuning on our side is involved. All rows of Table 3 are retrained by us in the same codebase with the same number of training steps, and the matched baselines in App. N are likewise our own runs under the same seeds, resets, and interaction budget. This is now stated in Sec. 4.1 and App. B.

Minor points.

  • Teacher forcing: yes, standard teacher-forced next-token cross-entropy; "autoregressively" referred to generation. Fixed.
  • Clip in r_end: the lower bound is redundant; we now write min(·, 1).
  • Milestone term: each milestone is rewarded at most once per episode (the stage is tracked as a running maximum), so re-reaching a stage after dropping the object earns nothing; the notation is fixed. You are right that the reward is not strictly potential-based. We removed that claim and now call it dense progress shaping, with the reasons given in App. C.3.
  • Baseline sources: stated in Sec. 4.1 and App. B (see Q7).
  • Fig. 3 caption: it now identifies the rows (top: action-only RL; bottom: TCoT-RL); the row labels in the App. K caption were swapped and are corrected.
  • C40, 20²/40², and first 5 / all in Fig. 1: defined in the caption and in Sec. 4.3.
  • Bhatia et al.: added.

Thank you for the detailed follow up. This does clarify some ambiguity.

Regarding Figure 1 (c), if I understand correctly, the blue and orange dots (which compare two different methods) both have similar y-values, and therefore similar trajectory coverage. However, the blue dots lie to the right of y=x, so they have more action coverage as well, at a given level of trajectory coverage. Is this not advantage to the baseline (blue)?

Reviewer_bSupOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TCoT-RL, a hierarchical reinforcement-learning framework for vision-language-action models. Instead of exploring only through low-level action noise, the method first samples a variable-length sequence of image-space end-effector waypoints as a high-level motion hypothesis, and then generates continuous actions conditioned on this trajectory. The trajectory and action policies are jointly optimized using separate group-relative advantages, with trajectory rewards based on task progress and stage-aware endpoint grounding. Experiments on LIBERO and LIBERO-Plus show improved performance and broader exploration compared with supervised and action-level RL baselines.

Critical strengths
  1. Strong motivation. The paper identifies a meaningful limitation of action-level exploration: local action perturbations may be inefficient for discovering alternative high-level intents in long-horizon manipulation. Introducing an explicit trajectory-level exploration space is a natural and well-motivated solution.
  2. Strong empirical performance. TCoT-RL achieves a reported success rate of 86.8% on LIBERO-plus.
Critical weaknesses
  1. Dependence on privileged simulator information and lack of real-robot validation. The core trajectory reward requires simulator-only object poses, task stages, goal geometry, camera calibration, and success predicates. Without real-robot experiments or a vision-only reward alternative, the practicality and transferability of the method remain unclear.
  2. The reward does not directly support the claimed exploration mechanism. TCoT-RL is motivated as a way to explore diverse high-level motion hypotheses, but its trajectory reward is largely based on task progress and endpoint proximity; trajectories reaching similar endpoints can therefore receive similar credit, even if they differ little in behavioral intent or collapse to the same route. The ablations show that endpoint shaping improves task success, but do not isolate whether the reported diversity gains arise from optimizing a genuinely diverse trajectory distribution or merely from endpoint-conditioned sampling.
  3. Entangled hierarchical credit assignment. The trajectory-level return includes progress measured after executing low-level actions, so the trajectory advantage remains affected by action-level stochasticity. Separate GRPO objectives do not provide counterfactual attribution between planning and execution.
Questions for authors

see weakness above

2 comments
AuthorsOpenReview ↗

Reviewer bSup

Thank you for the thoughtful critique.

W1 (privileged information, real robot). See G2. Privileged signals are used only for training-time rewards and never as policy inputs. TCoT-RL tolerates substantial noise in them (App. J), and an off-the-shelf VLM (zero-shot) localizes the endpoint targets with errors below the largest noise level we tested (median 10.5–16.7 px vs. ≈47 px). We acknowledge that real-robot post-training and end-to-end training with vision-derived rewards remain future work, and we have expanded the Limitations accordingly.

W2 (reward vs. the exploration mechanism). We agree that the reward does not explicitly encourage diversity, and we clarified this. Exploration comes from sampling a stochastic trajectory policy. The reward scores progress and where a plan ends, not how it gets there. The concern decomposes into two testable questions:

  1. Do same-endpoint routes produce different behavior, and are they viable? Yes (E2, G3). Changing only the route lowers success by up to 7.0 pp, so routes are not ignored. Yet routes the policy itself samples for the same endpoint keep success within 0.8 pp while the executed path deviates from the original by 25.2 px on average (22.8 px at mid-chunk). Exploration is therefore over genuinely different routes, not merely over endpoints.
  2. Does optimization collapse the distribution? No, not in our run (E8, new App. U). Among sampled trajectories with endpoints within 5 px, route dispersion stays at 96–102% of its SFT value over 24 RL updates (3.90 → 3.92 px), and the share of contexts with multiple distinct routes goes from 30% to 34%. The reward thus shapes where plans end, while the route diversity within an endpoint is preserved.

Sec. 4.3 isolates the exploration space (SFT checkpoint), while the Reward-shaped Action RL vs. Joint TCoT-RL contrast shows that the same reward helps only when its gradient reaches the trajectory policy.

W3 (entangled credit assignment). This is a fair point, and we quantified it (new E3). From 64 fixed simulator states (4 suites, three time indices), we sampled 8 trajectories per state and executed each 8 times with different flow noise. We then measured (new App. P) the share of reward variance explained by the trajectory choice (ICC), and whether trajectory rankings replicate across independent noise draws:

SignalICCRank consistency
Episode success (task return)0.009−0.11
Episode-accumulated trajectory return0.220.31
Per-chunk progress term0.230.31
Per-chunk trajectory reward (used)0.450.32

(ICC computed over states where the signal varies.)

The results support three conclusions:

  1. The reviewer is right that execution noise remains substantial. Even for the signal we use, more than half of the variance comes from execution. We now describe the scheme as variance reduction rather than attribution.
  2. The design still makes trajectory credit informative. The task outcome carries essentially no information about the trajectory choice. The trajectory reward raises the trajectory-attributable share by about 50×.
  3. Both design choices contribute. Crediting each trajectory with its immediate chunk reward, rather than the accumulated return, doubles the ICC (0.45 vs. 0.22). Adding the execution-independent endpoint term raises it from 0.23 to 0.45.

Note that these measurements restore identical states, which is cleaner than the same-reset, same-time-index baseline used in training (G4). A fully counterfactual estimator — averaging K executions per trajectory — would remove the remaining noise but multiplies rollout cost by K; we discuss this trade-off in App. P.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your thoughtful feedback. We would like to kindly ask whether our rebuttal has addressed your concerns. If you have any remaining questions or new concerns, we would be happy to discuss them further.

Reviewer_dCdzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TCoT-RL, a method that improves VLA exploration by learning both motion plans and their execution. A high-level policy generates image-space waypoints, while a low-level policy produces continuous actions conditioned on them. After supervised initialization, both policies are updated through RL, using separate learning signals based on task outcomes, progress, and trajectory endpoints. Experiments report 98.6% success on LIBERO and 89.6% on LIBERO-Plus, with the latter using LIBERO-Plus demonstrations during supervised training.

Critical strengths
  1. The paper addresses a meaningful problem with a clear motivation. Exploring through low-level actions can make coordinated changes in behavior difficult to discover. Treating trajectory generation as a trainable exploration policy offers an intuitive way to explore different approaches and routes.

  2. The controlled analyses provide useful support for the method. Reward-matched baselines help separate the contribution of trajectory optimization from simply adding denser rewards. Paired rollout measurements show broader spatial coverage and greater early-action diversity, while trajectory-input interventions demonstrate that the action model uses the generated plans.

  3. The method shows promising results with modest training overhead. Evaluations cover standard manipulation tasks and several visual and spatial changes in LIBERO-Plus. Under the matched training setup, TCoT-RL requires only 1.4% more online training time than the closest reward-matched action baseline, supporting its computational practicality in this setting.

Critical weaknesses
  1. The role of intermediate waypoints remains unclear. The waypoint-count comparisons and trajectory interventions show that trajectory information is useful. However, they do not fully separate the benefit of choosing a good route from simply pointing to the right endpoint. Testing different paths with the same endpoint would help clarify this distinction.

  2. The main performance gaps need stronger evidence across training runs. In Table 3, the full method improves over action-only RL by one percentage point. The appendix reports similar results for two training seeds of the full method, but comparable repeated runs are limited for the main baselines and ablations. The stability of these performance gaps therefore remains uncertain.

  3. Applicability beyond simulator-provided rewards remains untested. The dense rewards use object positions, task stages, and camera information. The authors clearly restrict this information to training and test noisy reward labels. However, how these rewards would be obtained in a real-world setting remains open.

  4. Some training details need clarification. The paper explains how delayed feedback is assigned to the corresponding trajectory, but does not fully explain which returns each prediction receives or how multiple predictions within an episode enter the group comparison. A concrete example would improve reproducibility.

Questions for authors
  1. Could the authors keep the endpoint fixed and change only the intermediate waypoints, then measure changes in actions and task success?

  2. Could the full method and the main action-only RL baseline be compared across the same training seeds, reporting the mean performance gap and its variation?

  3. How could the required rewards be constructed without simulator-provided object positions and task stages? Which signals would be hardest to estimate from vision?

  4. Does each trajectory prediction receive the current action chunk’s reward or accumulated future returns? Please clarify the group normalization, ideally with brief pseudocode, and provide the value of omega.

  5. Beyond overall training cost, what inference latency does trajectory generation add, and how does it affect the control rate?

2 comments
AuthorsOpenReview ↗

Reviewer dCdz

Thank you for the concrete questions. G1–G4 address seeds, rewards, waypoints, and credit assignment; details follow.

Q1 (fixed endpoint, different intermediates). See G3. With the endpoint fixed, unnatural routes lower success (straight −5.5 pp, mirrored −7.0 pp, the latter matching the drop from removing the trajectory entirely), whereas the model's own alternative routes do not (−0.8 pp). Intermediate waypoints therefore matter beyond the endpoint; with Donor routes, the executed path deviates from the original by 25.2 px on average (22.8 px at mid-chunk) while success is preserved.

Q2 (same seeds, mean gap and variation). See G1. TCoT-RL outperforms Action-only GRPO on all three shared seeds by +2.23 ± 0.49 pp, and the reward-matched Dense Action-GRPO by +1.53 ± 0.76 pp; the seed-to-seed std is 0.2–0.4 pp.

Q3 (rewards without simulator state; hardest signals). See G2. Our offline localization study gives a partial answer. The placement target is localized well (median 10.5 px, 86% within 20 px). The grasped object is harder (median 16.7 px, 67% within 20 px), plausibly because of occlusion by the gripper near contact. We expect the task stage — especially whether an object is grasped — to be the hardest signal, since it requires inferring contact state rather than location. App. J shows the method tolerates 30% stage corruption (−3.9 pp). Notably, the endpoint term — the localization-only part of the reward — is what separates full TCoT-RL from the progress-only variant (Table 7: 91.4% vs. 83.6% on 128 fixed episodes).

Q4 (returns, group normalization, ω). Each trajectory prediction receives the immediate reward of the chunk it conditions, not accumulated future returns. ω = 1.0. Pseudocode (new App. M):

for each RL update:
  32 (task, reset) groups × G = 8 parallel episodes; 480 steps = 48 chunks of 10 steps
  for episode i, chunk k:
      tau[i,k] ~ pi_traj(. | c[i,k]);  a[i,k] ~ pi_act(. | c[i,k], tau[i,k])
      execute 10 steps
      r_traj[i,k] = 0.10 * sum_{steps in chunk} r_prog + 0.05 * r_end(tau[i,k])

  # trajectory level: immediate reward, normalized per chunk index
  A_traj[i,k] = (r_traj[i,k] - mean_j r_traj[j,k]) / (std_j r_traj[j,k] + eps)
  tokens of tau[i,k] use A_traj[i,k]

  # action level: one return per episode, truncated at the first done
  J_act[i] = sum_{k <= done_i} (r_task[i,k] + omega * r_traj[i,k]),  omega = 1.0
  A_act[i] = (J_act[i] - mean_j J_act[j]) / (std_j J_act[j] + eps)
  denoising transitions of a[i,k], k <= done_i, use A_act[i]

The 48 predictions of an episode therefore each enter a separate group comparison at their own chunk index. Only at k = 0 do the 8 candidates share an identical observation; for k > 0, the baseline is a same-reset, same-time-index baseline. The original Eq. (5) and the phrase "accumulated trajectory-specific return" were imprecise, and we have corrected them.

Q5 (inference latency). On one A100 (batch 1), each 10-action chunk takes 735 ms on average (median 637 ms, p90 1318 ms):

  • trajectory generation (18.9 tokens on average): 558 ms (median 461 ms, p90 1140 ms);
  • 4-step flow action generation: 177 ms.

For comparison, π0.5, which generates only actions, takes 177 ms per chunk, so trajectory generation increases per-chunk latency by about 4.2×. With synchronous execution, this gives an inference-limited rate of 13.6 actions/s on average (7.6 at p90). The cost comes from serially decoding ≈19 tokens at batch size 1 (≈30 ms/token, with KV caching). At deployment, it can be largely hidden by generating the next trajectory asynchronously while the current chunk executes. We added these numbers to Sec. 4.1 and App. Q. The 1.4% figure in the paper refers to training wall-clock time, where the picture differs:

  • decoding is batched over 16 environments per GPU, which amortizes the per-token cost;
  • simulator stepping (≈680 s per update) and the rollout pipeline dominate each update;
  • policy inference accounts for only ≈15–20% of update time.

The batch-1 latency above is therefore the relevant number for deployment, not for training cost.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your thoughtful feedback. We would like to kindly ask whether our rebuttal has addressed your concerns. If you have any remaining questions or new concerns, we would be happy to discuss them further.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response for all reviewers and AC

We thank all reviewers for their careful reading. Reviewers agree the motivation is clear (dCdz, ZFHw, bSup), the controlled analyses are useful (dCdz, ZFHw), and the overhead is modest (dCdz). We address four shared concerns here (G1–G4); individual replies follow. Revisions in the PDF are marked in blue.

G1. Reliability across training runs (dCdz, ZFHw). We trained TCoT-RL and the two strongest matched action-level baselines with three seeds each (24 updates). All checkpoints are evaluated on the same 1,024 episodes (256 per suite; new App. N).

MethodSeed 42Seed 43Seed 44Mean ± std
Action-only GRPO83.1482.9682.7682.95 ± 0.19
Dense Action-GRPO (exec. endpoint)83.7483.9883.2283.65 ± 0.39
TCoT-RL85.2584.7785.5285.18 ± 0.38

TCoT-RL is best on every seed. The paired gaps are:

  • over Action-only GRPO: +2.23 ± 0.49 pp (per seed +2.11 / +1.81 / +2.76);
  • over the reward-matched Dense Action-GRPO: +1.53 ± 0.76 pp (+1.51 / +0.79 / +2.30).

Seed-to-seed variation (0.2–0.4 pp std) is several times smaller than the gaps. With three seeds, a seed-level paired t-test has little power (p = 0.015 and 0.072), so we present the paired gaps as evidence of a consistent effect rather than a significance claim. (This reduced protocol is not directly comparable with Table 3, which uses the official 10,030-variant evaluation after 60 updates.)

G2. Dependence on simulator-provided reward information (dCdz, ZFHw, bSup). We agree this is the main practical limitation. Three points:

  1. Privileged signals enter only the training-time reward, never the policy (App. J, Table 11). All RL baselines we compare against likewise rely on simulator success predicates.
  2. TCoT-RL is robust to noisy labels: endpoint noise up to σ = 40 px costs ≤ 3.1 pp, and 30% stage corruption costs 3.9 pp (Table 12).
  3. New: offline validation of vision-based localization (App. R). On 374 policy-rollout images (453 placement targets, 4 suites, 2 seeds), we localize the two stage-dependent endpoint targets in the policy's 224-px frame with an off-the-shelf VLM (Qwen3.5-9B, zero-shot, prompted with the task instruction).
    • Placement target: median error 10.5 px (p90 62 px, 86% within 20 px).
    • Grasped object: median error 16.7 px (p90 54 px, 67% within 20 px).
    Both error distributions are smaller, at the median and at p90, than the σ = 40 px endpoint-noise condition of App. J (isotropic Gaussian noise added in the same frame; radial median ≈ 47 px, p90 ≈ 86 px), under which TCoT-RL loses only 3.1 pp. VLM localization errors may be spatially correlated rather than i.i.d., so this is indirect evidence, and end-to-end training with VLM-derived rewards remains to be done.

Real-robot post-training remains future work and is now discussed in the Limitations.

AuthorsOpenReview ↗

General Response (Part 2/2)

G3. Does the route matter, or only the endpoint? (dCdz, bSup) Exploration comes from sampling a stochastic trajectory policy. The trajectory reward scores progress and where a plan ends, not its route, so it neither rewards nor penalizes routes for their own sake.

  • Same endpoint, different route (E2, new App. O). On 128 fixed resets, we keep each generated endpoint and replace only the intermediate waypoints at every chunk. Success (original: 117/128) changes as follows:
    • Donor — intermediates borrowed from another sample of the same context whose endpoint lies within 5 px of the original: 116/128 (−0.8 pp);
    • Straight — straight-line intermediates: 110/128 (−5.5 pp);
    • Mirrored — intermediates reflected across the start–end line: 108/128 (−7.0 pp).
    A mirrored route with the correct endpoint is as harmful as removing the trajectory altogether (masked: 108/128, Table 9). The action expert therefore uses route information beyond the endpoint. At the same time, the policy's own alternative routes to the same endpoint preserve success while changing the executed path (mean executed-path deviation from the original: 25.2 px, 22.8 px at mid-chunk). The trajectory distribution thus contains multiple viable, behaviorally distinct routes, which is the structure TCoT-RL explores over.
  • Diversity during RL (E8, new App. U). On 64 fixed simulator states, we sampled 32 trajectories per state from checkpoints after 0 (SFT), 6, 12, 18, and 24 updates, and measured route dispersion among trajectories whose endpoints lie within 5 px of each other. It stays at 96–102% of its SFT value throughout training (3.90 px → 3.92 px), and the share of contexts with multiple distinct routes rises slightly (30% → 34%). RL thus concentrates where plans end without collapsing how they get there.

We also clarify that Sec. 4.3 is performed at the SFT checkpoint and isolates the exploration space, whereas Table 3 isolates the effect of optimizing it.

G4. Precise credit assignment (dCdz, bSup, ZFHw). We have rewritten Sec. 3.4 to match the implementation exactly and added pseudocode (new App. M). Summary:

  • Each GRPO group consists of 8 parallel episodes from the same task/reset, each with 48 chunks.
  • Every chunk samples its own trajectory. That trajectory receives the immediate trajectory reward of the chunk it conditions — not accumulated future returns — normalized across the 8 episodes at the same chunk index.
  • The action policy receives one advantage per episode, computed from Σ_{k≤done}(r_task + ω r_traj) with ω = 1.0.

The original text ("accumulated trajectory-specific return"; groups sharing an identical c_t) was imprecise. Only the first chunk shares an identical observation; later chunks use a same-reset, same-time-index baseline. A new variance decomposition (E3, new App. P; see the reply to bSup) shows that this immediate per-chunk credit is about twice as trajectory-attributable as the accumulated return (ICC 0.45 vs. 0.22).

Summary of revisions (blue in the PDF). Original table, equation, and appendix numbers are unchanged; new material is in App. M–U. App. K now keeps three qualitative figures, so the former Figs. 14–15 are Figs. 9–10.

  • Sec. 3.4 + App. M — Credit assignment rewritten to match the implementation: per-chunk immediate trajectory reward normalized per chunk index; one episode-level action advantage truncated at the first success; ω = 1.0; pseudocode. (dCdz Q4, bSup W3, ZFHw Q1)
  • Sec. 3.3 + App. C.3 — Milestone written as a running maximum M_t; "potential-based" claim removed and the reasons stated; r_end uses min(·, 1); teacher forcing clarified. (ZFHw minor)
  • Sec. 4.3–4.4 + App. N, S — Three seeds for TCoT-RL and the two strongest matched baselines, with paired gaps; Action-only GRPO retrained with doubled action noise does not improve (83.6% vs. 85.2%; TCoT-RL 91.4% on the same episodes). (dCdz W2/Q2, ZFHw W1, E6)
  • Sec. 4.3–4.4 + App. O, U — Fixed-endpoint route intervention (Donor / Straight / Mirrored); route diversity among same-endpoint trajectories preserved over RL (96–102% of SFT). (dCdz Q1, bSup W2)
  • Sec. 3.4, Table 3 + App. P, T — Variance decomposition (ICC) of task and trajectory signals; trajectory-only RL added to Table 3 (86.1); task reward in the trajectory objective gives no gain (86.5 vs. 86.8). (bSup W3, ZFHw Q1/Q3)
  • Sec. 4.1 + App. Q — Inference latency vs. π0.5. (dCdz Q5)
  • Sec. 5 + App. R — Expanded Limitations (incl. no comparison with intrinsic-motivation bonuses such as RND); offline VLM localization compared with the reward-noise study. (dCdz Q3, ZFHw W3, bSup W1)
  • Sec. 4.4 (waypoints), Table 1, Figs. 1 and 3, Sec. 4.3, Related Work — Weaker waypoint-allocation claim; Table 1 split by SFT setting; caption clarifications; baseline sources; SFT-checkpoint scope of Sec. 4.3; SARL (Bhatia et al.) added. (ZFHw Q4–Q6, minor)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.